I recently met for coffee with two attorneys: a senior attorney at a prominent Big Law litigation firm and a managing attorney at a midsize commercial real estate firm. I asked each what they consider when hiring new attorneys, what concerns they have about those attorneys’ abilities, and how their firms approach artificial intelligence.
Their responses conveyed the same expectation: new hires must be able to use AI competently, ethically, and efficiently. One firm even evaluates applicants’ AI skills during the hiring process.
Those conversations highlighted our challenge as legal educators. We must prepare students to use AI as a professional advantage while ensuring that they develop the knowledge and judgment necessary to practice law responsibly. The ABA’s Formal Opinion 512 reinforces the importance of understanding AI’s capabilities and limitations and appropriately reviewing its output.[1] Students need the legal foundation to meet those responsibilities.
The challenge becomes particularly acute during bar preparation. Students face a compressed study schedule, financial pressures, and an examination that can determine when they enter practice. Research on academic misconduct identifies pressures and incentives that can contribute to students’ decisions to cheat.[2] Although that research does not establish the prevalence of AI misuse during bar preparation, it offers a framework for understanding why some students may use AI to bypass essential practice.
It is clear that AI advances professional preparation and yet also interferes with the learning that preparation requires. In approaching this problem, there are three points to consider: (i) The purpose of the task should determine the rules, (ii) licensing advancements could evaluate Ai competence, and (iii) bar preparation must preserve meaningful practice while also promoting improved assessments.
- The Purpose of the Task Should Determine the Rules regarding Ai use.
A student who uses AI during a hiring exercise expressly designed to evaluate that skill demonstrates a relevant professional ability. A student who uses AI to generate a practice essay intended to develop independent legal reasoning may avoid the very work the exercise requires.
The purpose and conditions of the task determine whether assistance is appropriate.
That distinction should guide legal education and licensing. Students need opportunities to learn how to use AI, including how to identify inaccurate rules, unsupported conclusions, and missing issues. They also need opportunities to develop and demonstrate their own reasoning without AI supplying the analysis.
These objectives reinforce each other. A student who lacks foundational legal knowledge may struggle to recognize an unreliable AI-generated answer. Conversely, a student who reasons effectively without AI has not necessarily demonstrated the ability to use it responsibly.
An assessment should therefore make clear what it measures. Does it evaluate independent issue identification and analysis? Does it evaluate the ability to verify and improve AI-generated work? Or does it assess both through separate components? Without that clarity, a polished submission may reveal little about the student’s actual capabilities.
- Professional licensing could assess Ai competence.
If jurisdictions award licenses based, in part, on demonstrated competence, should the licensing process also evaluate responsible AI use?
The attorneys I spoke with already consider that ability relevant to hiring. Professional expectations also require lawyers using AI to exercise judgment, protect confidential information, verify output, and accept responsibility for their work. An examination that excludes AI throughout cannot directly establish whether an applicant can fulfill those responsibilities while using it.
AI competence could be evaluated through a separate licensing assessment, a designated bar examination component, or an updated professional responsibility examination. A supervised exercise might require applicants to identify errors in AI-generated analysis, verify authorities, and explain whether the work is suitable for a client matter.
The appropriate format remains open to debate. Any approach would require careful attention to fairness, access, and what the assessment actually demonstrates. But excluding AI from some assessments need not mean excluding AI competence from the licensing process altogether.
Licensing authorities should consider how to evaluate both independent legal reasoning and responsible technological assistance. Each provides different evidence of readiness to practice.
- Bar preparation must preserve meaningful practice and promote improved assessments.
Commercial bar preparation providers face a related challenge: ensuring that their courses promote learning and that their progress measures accurately communicate what students have accomplished.
Consider a hypothetical student approaching midnight with two practice essays remaining. Writing independently will take time and may expose weaknesses. An AI tool can generate organized responses in seconds. The student reads those responses, recognizes the rules, and marks the assignments complete.
The dashboard reflects progress. Whether the student has improved at identifying issues, retrieving rules, or applying facts remains uncertain.
This ambiguity matters when students, faculty, or coaches use completion data to assess readiness. A completion marker does not necessarily reveal how an assignment was completed. Even a favorable practice score may provide limited evidence of independent ability if AI supplied the central reasoning.
Students may also mistake familiarity for mastery. An answer can seem obvious after someone—or something—has organized the issues and explained the rules. Producing that analysis independently on a new problem requires a different level of understanding.
Providers should therefore distinguish between assisted learning activities, assignments requiring an independent initial attempt, and diagnostic assessments intended to approximate examination conditions. Their progress reports should reflect those distinctions rather than imply that all completed work supplies equivalent evidence of readiness.
Monitoring technology may support assessment integrity, but detection alone cannot establish competence. Providers can nevertheless invest in assessment practices that produce stronger evidence of learning. Periodic independent timed exercises protected testing environments, and opportunities for students to explain their reasoning can supplement routine course data.
An effective sequence might begin with an independent essay attempt, followed by authorized AI feedback and revision. The student could then explain the changes and complete a later lockdown independent exercise on a new problem. This approach permits assistance while preserving opportunities to determine whether learning transfers.
Commercial providers have an opportunity to advance these practices through their course designs, technology investments, and communications about readiness.
Finally, should we use AI as the catalyst to redefine readiness?
AI invites a broader discussion about attorney licensing. Should the profession place greater weight on supervised, practice-based demonstrations of competence? Could aspects of clinical training in medicine inform an approach that evaluates performance over time?
Those questions deserve consideration. They do not eliminate the immediate responsibility to improve existing preparation and assessment systems, but it does promote more “on the job” training akin to the medical profession.
Law schools, licensing authorities, and bar preparation providers should align their expectations around demonstrable abilities. Students should know when they must work independently, when assistance is appropriate, and how each activity prepares them for examination or practice.
Our preparation systems should make that evidence visible.
(Amy Vaughan-Thomas)
[1] American Bar Association, Formal Opinion 512: Generative Artificial Intelligence Tools (July 29, 2024), available here. The opinion addresses lawyers’ ethical responsibilities when using generative AI.
[2] Miles PJ, Campbell M, Ruxton GD. “Why Students Cheat and How Understanding This Can Help Reduce the Frequency of Academic Misconduct in Higher Education: A Literature Review.” Journal of Undergraduate Neuroscience Education. 2022;20(2)–A160. doi: 10.59390/LXMJ2920. PMID: 38323052; PMCID: PMC10653228.
