{"id":85344,"topic":"ai","source":"MedTech Dive","title":"Cognita Imaging wins FDA contract to test LLMs in evaluating radiology AI - MedTech Dive","url":"https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","url_hash":"15fd9268d670d824e0fbd48c75c848eb3a2a1f67","author":"","summary":"Cognita Imaging wins FDA contract to test LLMs in evaluating radiology AI MedTech Dive","content":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.\nThe contract, which started on June 22 and will run for 18 months, is intended to address a core problem with generative AI. Currently, most tools are evaluated against a panel of radiologist experts, but for models that can evaluate a wide variety of cases and generate different outputs, this becomes more challenging and time-consuming.\nToday, most of the AI tools authorized by the FDA are in radiology. Many of them are one-off solutions, such as to detect certain blood clots in a head CT scan, or a collapsed lung in a chest X-ray. Evaluating these types of models is straightforward, said Akshay Chaudhari, Cognita co-founder and an associate professor of radiology and biomedical data science at Stanford University. However, tools that only identify one or two things are not very helpful in radiologist practices.\nNewer tools being developed by Cognita and its peers look to generate full text radiology reports.\n“But it's more open-ended in nature, so there could be hundreds of findings that could potentially be represented in this output,” Chaudhari said. “So, how do you meaningfully evaluate these hundreds of findings?”\nCognita is working with the Center for Devices and Radiological Health’s regulatory science team to test the approach of using multiple LLMs to review the output of AI radiology models, with the goal of being comfortable with their understanding of the benefits and risks of those tools. The company will build and test the framework, and then apply it to 1 million patient exams from a large U.S. cohort. Researchers will examine the model’s performance across different patient groups, care settings, imaging equipment and diseases, including rare findings.\nWhen there are disagreements that have clinical significance, radiologists will be brought in to see if the problem came from the AI-generated report, the LLM jury or the original radiologist’s report.\nTwo concerns with generative AI are the potential for hallucinations, or information that seems plausible but is incorrect, and omissions, when a model does not include something that should be there. Chaudhari said he intends to be able to quantify the rate of hallucinations and omissions and whether they were clinically meaningful through this approach.\nCognita will provide the FDA with a report on the project’s findings and limitations, as well as the software code, guidance for building LLM juries, analyses comparing large and small validation cohorts, and radiologist-reviewed discrepancy studies.\nChaudhari said he wants the learnings from the test to be generalizable, so that the FDA may use them in regulations and guidance.\nFounded in 2024, Cognita was sold a year later to Radiology Partners, the largest radiology practice in the U.S. Chaudhari said having feedback from those radiologists about what challenges they’re facing, across diverse practice areas from an academic medical center to an outpatient community facility, has been helpful.\nCurrently, the U.S. faces a shortfall of radiologists, resulting in longer work hours and larger case backlogs.\nBreakthrough device designation\nCognita is working with the FDA on a separate initiative to develop a vision-language model that can interpret chest X-rays and generate reports, which are then reviewed by a radiologist. The company received the FDA’s breakthrough designation earlier this year for the device, which has not been authorized by the agency yet.\nChaudhari said the breakthrough designation is separate from the company’s research contract. Cognita is working with the FDA to determine what type of clinical study the company could use to evaluate the accuracy, benefits and risks of its vision-language model.\n“There's still work to be done in developing the exact study plan and actually implementing the study, but we're confident that through these very nice partnerships that the FDA fosters that there is a path forward, because there is a real clinical need for these tools,” Chaudhari said.\nBetween the breakthrough designations, the research contracts and a discussion paper shared by the agency last month, the FDA is taking a closer look at how to regulate devices that use generative AI.","image_url":"https://imgproxy.divecdn.com/krk4perQZTr9pTTnNI71djq6AZzFcdstlWcEjBnbv-c/g:ce/rs:fit:770:435/Z3M6Ly9kaXZlc2l0ZS1zdG9yYWdlL2RpdmVpbWFnZS9SYWRpb2xvZ2lzdC5qcGc=.webp","lang":"en","published_at":"2026-09-18T15:52:39+00:00","fetched_at":"2026-09-18T16:15:04+00:00","status":"read","starred":0,"extract_state":"ok","summary_auto":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.","cluster_id":null,"extract_retries":0,"extract_error":null,"contract_version":"news_item.v1","format_contract_version":"news_item_formats.v1","dedup_url":"https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","quality_profile":{"profile_version":"extraction_quality.v2","bucket":"high","confidence":0.9,"failure_kind":"none","retryable":false,"retry_after_attempts":0,"reason":"High confidence: full text extraction produced 4528 characters.","operator_guidance":{"severity":"ok","recommended_action":"trust_full_text","next_step":"Use the extracted full text as the primary article source.","operator_label":"Ready","can_retry":false,"can_use_summary":false,"diagnostics_required":false},"content_depth":{"contract_version":"content_depth.v1","category":"full_text","label":"Full text","has_full_text":true,"has_summary":true,"content_length":4528,"summary_length":341,"usable_text_length":4528,"source_field":"content"},"legacy_collapsed":false,"signals":{"extract_state":"ok","extract_error":null,"extract_retries":0,"content_length":4528,"summary_length":341}},"news_item":{"id":85344,"canonical_url":"https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","source_url":"https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","title":"Cognita Imaging wins FDA contract to test LLMs in evaluating radiology AI - MedTech Dive","source_name":"MedTech Dive","author":null,"published_at":"2026-09-18T15:52:39+00:00","locale":"en","topic":"ai","tags":[],"rss_summary":"Cognita Imaging wins FDA contract to test LLMs in evaluating radiology AI MedTech Dive","full_text":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.\nThe contract, which started on June 22 and will run for 18 months, is intended to address a core problem with generative AI. Currently, most tools are evaluated against a panel of radiologist experts, but for models that can evaluate a wide variety of cases and generate different outputs, this becomes more challenging and time-consuming.\nToday, most of the AI tools authorized by the FDA are in radiology. Many of them are one-off solutions, such as to detect certain blood clots in a head CT scan, or a collapsed lung in a chest X-ray. Evaluating these types of models is straightforward, said Akshay Chaudhari, Cognita co-founder and an associate professor of radiology and biomedical data science at Stanford University. However, tools that only identify one or two things are not very helpful in radiologist practices.\nNewer tools being developed by Cognita and its peers look to generate full text radiology reports.\n“But it's more open-ended in nature, so there could be hundreds of findings that could potentially be represented in this output,” Chaudhari said. “So, how do you meaningfully evaluate these hundreds of findings?”\nCognita is working with the Center for Devices and Radiological Health’s regulatory science team to test the approach of using multiple LLMs to review the output of AI radiology models, with the goal of being comfortable with their understanding of the benefits and risks of those tools. The company will build and test the framework, and then apply it to 1 million patient exams from a large U.S. cohort. Researchers will examine the model’s performance across different patient groups, care settings, imaging equipment and diseases, including rare findings.\nWhen there are disagreements that have clinical significance, radiologists will be brought in to see if the problem came from the AI-generated report, the LLM jury or the original radiologist’s report.\nTwo concerns with generative AI are the potential for hallucinations, or information that seems plausible but is incorrect, and omissions, when a model does not include something that should be there. Chaudhari said he intends to be able to quantify the rate of hallucinations and omissions and whether they were clinically meaningful through this approach.\nCognita will provide the FDA with a report on the project’s findings and limitations, as well as the software code, guidance for building LLM juries, analyses comparing large and small validation cohorts, and radiologist-reviewed discrepancy studies.\nChaudhari said he wants the learnings from the test to be generalizable, so that the FDA may use them in regulations and guidance.\nFounded in 2024, Cognita was sold a year later to Radiology Partners, the largest radiology practice in the U.S. Chaudhari said having feedback from those radiologists about what challenges they’re facing, across diverse practice areas from an academic medical center to an outpatient community facility, has been helpful.\nCurrently, the U.S. faces a shortfall of radiologists, resulting in longer work hours and larger case backlogs.\nBreakthrough device designation\nCognita is working with the FDA on a separate initiative to develop a vision-language model that can interpret chest X-rays and generate reports, which are then reviewed by a radiologist. The company received the FDA’s breakthrough designation earlier this year for the device, which has not been authorized by the agency yet.\nChaudhari said the breakthrough designation is separate from the company’s research contract. Cognita is working with the FDA to determine what type of clinical study the company could use to evaluate the accuracy, benefits and risks of its vision-language model.\n“There's still work to be done in developing the exact study plan and actually implementing the study, but we're confident that through these very nice partnerships that the FDA fosters that there is a path forward, because there is a real clinical need for these tools,” Chaudhari said.\nBetween the breakthrough designations, the research contracts and a discussion paper shared by the agency last month, the FDA is taking a closer look at how to regulate devices that use generative AI.","excerpt":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.","extraction":{"state":"ok","confidence":0.9,"error":null,"explanation":"High confidence: full text extraction produced 4528 characters.","diagnostics_url":"/api/diagnose?url=https%3A//www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","quality_profile":{"profile_version":"extraction_quality.v2","bucket":"high","confidence":0.9,"failure_kind":"none","retryable":false,"retry_after_attempts":0,"reason":"High confidence: full text extraction produced 4528 characters.","operator_guidance":{"severity":"ok","recommended_action":"trust_full_text","next_step":"Use the extracted full text as the primary article source.","operator_label":"Ready","can_retry":false,"can_use_summary":false,"diagnostics_required":false},"content_depth":{"contract_version":"content_depth.v1","category":"full_text","label":"Full text","has_full_text":true,"has_summary":true,"content_length":4528,"summary_length":341,"usable_text_length":4528,"source_field":"content"},"legacy_collapsed":false,"signals":{"extract_state":"ok","extract_error":null,"extract_retries":0,"content_length":4528,"summary_length":341}}},"display_formats":["compact","card","full","digest_section","json"]},"daily_stack_record":{"title":"Cognita Imaging wins FDA contract to test LLMs in evaluating radiology AI - MedTech Dive","url":"https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","summary":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.","source":"MedTech Dive","date":"2026-09-18T15:52:39+00:00","content":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.\nThe contract, which started on June 22 and will run for 18 months, is intended to address a core problem with generative AI. Currently, most tools are evaluated against a panel of radiologist experts, but for models that can evaluate a wide variety of cases and generate different outputs, this becomes more challenging and time-consuming.\nToday, most of the AI tools authorized by the FDA are in radiology. Many of them are one-off solutions, such as to detect certain blood clots in a head CT scan, or a collapsed lung in a chest X-ray. Evaluating these types of models is straightforward, said Akshay Chaudhari, Cognita co-founder and an associate professor of radiology and biomedical data science at Stanford University. However, tools that only identify one or two things are not very helpful in radiologist practices.\nNewer tools being developed by Cognita and its peers look to generate full text radiology reports.\n“But it's more open-ended in nature, so there could be hundreds of findings that could potentially be represented in this output,” Chaudhari said. “So, how do you meaningfully evaluate these hundreds of findings?”\nCognita is working with the Center for Devices and Radiological Health’s regulatory science team to test the approach of using multiple LLMs to review the output of AI radiology models, with the goal of being comfortable with their understanding of the benefits and risks of those tools. The company will build and test the framework, and then apply it to 1 million patient exams from a large U.S. cohort. Researchers will examine the model’s performance across different patient groups, care settings, imaging equipment and diseases, including rare findings.\nWhen there are disagreements that have clinical significance, radiologists will be brought in to see if the problem came from the AI-generated report, the LLM jury or the original radiologist’s report.\nTwo concerns with generative AI are the potential for hallucinations, or information that seems plausible but is incorrect, and omissions, when a model does not include something that should be there. Chaudhari said he intends to be able to quantify the rate of hallucinations and omissions and whether they were clinically meaningful through this approach.\nCognita will provide the FDA with a report on the project’s findings and limitations, as well as the software code, guidance for building LLM juries, analyses comparing large and small validation cohorts, and radiologist-reviewed discrepancy studies.\nChaudhari said he wants the learnings from the test to be generalizable, so that the FDA may use them in regulations and guidance.\nFounded in 2024, Cognita was sold a year later to Radiology Partners, the largest radiology practice in the U.S. Chaudhari said having feedback from those radiologists about what challenges they’re facing, across diverse practice areas from an academic medical center to an outpatient community facility, has been helpful.\nCurrently, the U.S. faces a shortfall of radiologists, resulting in longer work hours and larger case backlogs.\nBreakthrough device designation\nCognita is working with the FDA on a separate initiative to develop a vision-language model that can interpret chest X-rays and generate reports, which are then reviewed by a radiologist. The company received the FDA’s breakthrough designation earlier this year for the device, which has not been authorized by the agency yet.\nChaudhari said the breakthrough designation is separate from the company’s research contract. Cognita is working with the FDA to determine what type of clinical study the company could use to evaluate the accuracy, benefits and risks of its vision-language model.\n“There's still work to be done in developing the exact study plan and actually implementing the study, but we're confident that through these very nice partnerships that the FDA fosters that there is a path forward, because there is a real clinical need for these tools,” Chaudhari said.\nBetween the breakthrough designations, the research contracts and a discussion paper shared by the agency last month, the FDA is taking a closer look at how to regulate devices that use generative AI.","confidence":0.9,"diagnostics_url":"/api/diagnose?url=https%3A//www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","quality_bucket":"high","failure_kind":"none","retryable":false,"quality_reason":"High confidence: full text extraction produced 4528 characters.","quality_profile":{"profile_version":"extraction_quality.v2","bucket":"high","confidence":0.9,"failure_kind":"none","retryable":false,"retry_after_attempts":0,"reason":"High confidence: full text extraction produced 4528 characters.","operator_guidance":{"severity":"ok","recommended_action":"trust_full_text","next_step":"Use the extracted full text as the primary article source.","operator_label":"Ready","can_retry":false,"can_use_summary":false,"diagnostics_required":false},"content_depth":{"contract_version":"content_depth.v1","category":"full_text","label":"Full text","has_full_text":true,"has_summary":true,"content_length":4528,"summary_length":341,"usable_text_length":4528,"source_field":"content"},"legacy_collapsed":false,"signals":{"extract_state":"ok","extract_error":null,"extract_retries":0,"content_length":4528,"summary_length":341}},"tags":[]},"fallback_formats":["markdown","json","html"],"actions":{"read":"/item/85344","export_markdown":"/api/items/85344/export?format=markdown","export_json":"/api/items/85344/export?format=json","diagnose":"/api/diagnose?url=https%3A//www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/"},"formats":{"full":{"id":85344,"title":"Cognita Imaging wins FDA contract to test LLMs in evaluating radiology AI - MedTech Dive","url":"https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","source":"MedTech Dive","author":null,"published_at":"2026-09-18T15:52:39+00:00","locale":"en","topic":"ai","tags":[],"excerpt":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.","full_text":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.\nThe contract, which started on June 22 and will run for 18 months, is intended to address a core problem with generative AI. Currently, most tools are evaluated against a panel of radiologist experts, but for models that can evaluate a wide variety of cases and generate different outputs, this becomes more challenging and time-consuming.\nToday, most of the AI tools authorized by the FDA are in radiology. Many of them are one-off solutions, such as to detect certain blood clots in a head CT scan, or a collapsed lung in a chest X-ray. Evaluating these types of models is straightforward, said Akshay Chaudhari, Cognita co-founder and an associate professor of radiology and biomedical data science at Stanford University. However, tools that only identify one or two things are not very helpful in radiologist practices.\nNewer tools being developed by Cognita and its peers look to generate full text radiology reports.\n“But it's more open-ended in nature, so there could be hundreds of findings that could potentially be represented in this output,” Chaudhari said. “So, how do you meaningfully evaluate these hundreds of findings?”\nCognita is working with the Center for Devices and Radiological Health’s regulatory science team to test the approach of using multiple LLMs to review the output of AI radiology models, with the goal of being comfortable with their understanding of the benefits and risks of those tools. The company will build and test the framework, and then apply it to 1 million patient exams from a large U.S. cohort. Researchers will examine the model’s performance across different patient groups, care settings, imaging equipment and diseases, including rare findings.\nWhen there are disagreements that have clinical significance, radiologists will be brought in to see if the problem came from the AI-generated report, the LLM jury or the original radiologist’s report.\nTwo concerns with generative AI are the potential for hallucinations, or information that seems plausible but is incorrect, and omissions, when a model does not include something that should be there. Chaudhari said he intends to be able to quantify the rate of hallucinations and omissions and whether they were clinically meaningful through this approach.\nCognita will provide the FDA with a report on the project’s findings and limitations, as well as the software code, guidance for building LLM juries, analyses comparing large and small validation cohorts, and radiologist-reviewed discrepancy studies.\nChaudhari said he wants the learnings from the test to be generalizable, so that the FDA may use them in regulations and guidance.\nFounded in 2024, Cognita was sold a year later to Radiology Partners, the largest radiology practice in the U.S. Chaudhari said having feedback from those radiologists about what challenges they’re facing, across diverse practice areas from an academic medical center to an outpatient community facility, has been helpful.\nCurrently, the U.S. faces a shortfall of radiologists, resulting in longer work hours and larger case backlogs.\nBreakthrough device designation\nCognita is working with the FDA on a separate initiative to develop a vision-language model that can interpret chest X-rays and generate reports, which are then reviewed by a radiologist. The company received the FDA’s breakthrough designation earlier this year for the device, which has not been authorized by the agency yet.\nChaudhari said the breakthrough designation is separate from the company’s research contract. Cognita is working with the FDA to determine what type of clinical study the company could use to evaluate the accuracy, benefits and risks of its vision-language model.\n“There's still work to be done in developing the exact study plan and actually implementing the study, but we're confident that through these very nice partnerships that the FDA fosters that there is a path forward, because there is a real clinical need for these tools,” Chaudhari said.\nBetween the breakthrough designations, the research contracts and a discussion paper shared by the agency last month, the FDA is taking a closer look at how to regulate devices that use generative AI.","reading_time_min":4,"extraction":{"state":"ok","confidence":0.9,"error":null,"explanation":"High confidence: full text extraction produced 4528 characters.","diagnostics_url":"/api/diagnose?url=https%3A//www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","quality_profile":{"profile_version":"extraction_quality.v2","bucket":"high","confidence":0.9,"failure_kind":"none","retryable":false,"retry_after_attempts":0,"reason":"High confidence: full text extraction produced 4528 characters.","operator_guidance":{"severity":"ok","recommended_action":"trust_full_text","next_step":"Use the extracted full text as the primary article source.","operator_label":"Ready","can_retry":false,"can_use_summary":false,"diagnostics_required":false},"content_depth":{"contract_version":"content_depth.v1","category":"full_text","label":"Full text","has_full_text":true,"has_summary":true,"content_length":4528,"summary_length":341,"usable_text_length":4528,"source_field":"content"},"legacy_collapsed":false,"signals":{"extract_state":"ok","extract_error":null,"extract_retries":0,"content_length":4528,"summary_length":341}}},"quality_profile":{"profile_version":"extraction_quality.v2","bucket":"high","confidence":0.9,"failure_kind":"none","retryable":false,"retry_after_attempts":0,"reason":"High confidence: full text extraction produced 4528 characters.","operator_guidance":{"severity":"ok","recommended_action":"trust_full_text","next_step":"Use the extracted full text as the primary article source.","operator_label":"Ready","can_retry":false,"can_use_summary":false,"diagnostics_required":false},"content_depth":{"contract_version":"content_depth.v1","category":"full_text","label":"Full text","has_full_text":true,"has_summary":true,"content_length":4528,"summary_length":341,"usable_text_length":4528,"source_field":"content"},"legacy_collapsed":false,"signals":{"extract_state":"ok","extract_error":null,"extract_retries":0,"content_length":4528,"summary_length":341}},"actions":{"read":"/item/85344","export_markdown":"/api/items/85344/export?format=markdown","export_json":"/api/items/85344/export?format=json","diagnose":"/api/diagnose?url=https%3A//www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/"}},"digest":{"id":85344,"title":"Cognita Imaging wins FDA contract to test LLMs in evaluating radiology AI - MedTech Dive","url":"https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","source":"MedTech Dive","topic":"ai","published_at":"2026-09-18T15:52:39+00:00","excerpt":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another…","quality_bucket":"high","quality_reason":"High confidence: full text extraction produced 4528 characters.","reading_time_min":4,"cluster_id":null},"card":{"display_title":"Cognita Imaging wins FDA contract to test LLMs in evaluating radiology AI - MedTech Dive","subtitle":"MedTech Dive · 2026-09-18","summary":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess…","badges":["quality:high"],"links":{"read":"/item/85344","original":"https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","diagnose":"/api/diagnose?url=https%3A//www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/"},"quality_warning":null},"export":{"title":"Cognita Imaging wins FDA contract to test LLMs in evaluating radiology AI - MedTech Dive","url":"https://www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","summary":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.","source":"MedTech Dive","date":"2026-09-18T15:52:39+00:00","content":"Cognita Imaging recently received a $1.29 million research contract from the Food and Drug Administration to test a new method of evaluating artificial intelligence-generated radiology reports. The goal is to assess whether a “jury” of large language models can evaluate another AI model’s report, and which cases need radiologist oversight.\nThe contract, which started on June 22 and will run for 18 months, is intended to address a core problem with generative AI. Currently, most tools are evaluated against a panel of radiologist experts, but for models that can evaluate a wide variety of cases and generate different outputs, this becomes more challenging and time-consuming.\nToday, most of the AI tools authorized by the FDA are in radiology. Many of them are one-off solutions, such as to detect certain blood clots in a head CT scan, or a collapsed lung in a chest X-ray. Evaluating these types of models is straightforward, said Akshay Chaudhari, Cognita co-founder and an associate professor of radiology and biomedical data science at Stanford University. However, tools that only identify one or two things are not very helpful in radiologist practices.\nNewer tools being developed by Cognita and its peers look to generate full text radiology reports.\n“But it's more open-ended in nature, so there could be hundreds of findings that could potentially be represented in this output,” Chaudhari said. “So, how do you meaningfully evaluate these hundreds of findings?”\nCognita is working with the Center for Devices and Radiological Health’s regulatory science team to test the approach of using multiple LLMs to review the output of AI radiology models, with the goal of being comfortable with their understanding of the benefits and risks of those tools. The company will build and test the framework, and then apply it to 1 million patient exams from a large U.S. cohort. Researchers will examine the model’s performance across different patient groups, care settings, imaging equipment and diseases, including rare findings.\nWhen there are disagreements that have clinical significance, radiologists will be brought in to see if the problem came from the AI-generated report, the LLM jury or the original radiologist’s report.\nTwo concerns with generative AI are the potential for hallucinations, or information that seems plausible but is incorrect, and omissions, when a model does not include something that should be there. Chaudhari said he intends to be able to quantify the rate of hallucinations and omissions and whether they were clinically meaningful through this approach.\nCognita will provide the FDA with a report on the project’s findings and limitations, as well as the software code, guidance for building LLM juries, analyses comparing large and small validation cohorts, and radiologist-reviewed discrepancy studies.\nChaudhari said he wants the learnings from the test to be generalizable, so that the FDA may use them in regulations and guidance.\nFounded in 2024, Cognita was sold a year later to Radiology Partners, the largest radiology practice in the U.S. Chaudhari said having feedback from those radiologists about what challenges they’re facing, across diverse practice areas from an academic medical center to an outpatient community facility, has been helpful.\nCurrently, the U.S. faces a shortfall of radiologists, resulting in longer work hours and larger case backlogs.\nBreakthrough device designation\nCognita is working with the FDA on a separate initiative to develop a vision-language model that can interpret chest X-rays and generate reports, which are then reviewed by a radiologist. The company received the FDA’s breakthrough designation earlier this year for the device, which has not been authorized by the agency yet.\nChaudhari said the breakthrough designation is separate from the company’s research contract. Cognita is working with the FDA to determine what type of clinical study the company could use to evaluate the accuracy, benefits and risks of its vision-language model.\n“There's still work to be done in developing the exact study plan and actually implementing the study, but we're confident that through these very nice partnerships that the FDA fosters that there is a path forward, because there is a real clinical need for these tools,” Chaudhari said.\nBetween the breakthrough designations, the research contracts and a discussion paper shared by the agency last month, the FDA is taking a closer look at how to regulate devices that use generative AI.","confidence":0.9,"diagnostics_url":"/api/diagnose?url=https%3A//www.medtechdive.com/news/cognita-imaging-wins-fda-contract-to-test-llms-in-evaluating-radiology-ai/830788/","quality_bucket":"high","failure_kind":"none","retryable":false,"quality_reason":"High confidence: full text extraction produced 4528 characters.","quality_profile":{"profile_version":"extraction_quality.v2","bucket":"high","confidence":0.9,"failure_kind":"none","retryable":false,"retry_after_attempts":0,"reason":"High confidence: full text extraction produced 4528 characters.","operator_guidance":{"severity":"ok","recommended_action":"trust_full_text","next_step":"Use the extracted full text as the primary article source.","operator_label":"Ready","can_retry":false,"can_use_summary":false,"diagnostics_required":false},"content_depth":{"contract_version":"content_depth.v1","category":"full_text","label":"Full text","has_full_text":true,"has_summary":true,"content_length":4528,"summary_length":341,"usable_text_length":4528,"source_field":"content"},"legacy_collapsed":false,"signals":{"extract_state":"ok","extract_error":null,"extract_retries":0,"content_length":4528,"summary_length":341}},"tags":[],"format_contract_version":"news_item_formats.v1"}}}