Llama Guard

Shieldstral is a 3B open-weights, policy-adaptive multimodal safety classifier designed to evaluate text, images, and text-plus-image content using policies defined at inference time. Instead of relying on a fixed taxonomy of harm categories, it frames moderation as a binary question-answering task: users provide an instruction describing the evaluation context and strictness, a yes-or-no safety question, and the content to judge. The model reads the “yes” and “no” logits and converts them into a continuous, calibrated safety score, allowing applications to threshold or rank results by confidence rather than depend on a single discrete label. This formulation unifies prompt classification, response moderation, refusal detection, toxicity detection, and multimodal safety in one interface, while letting teams adapt policies without retraining the model. Shieldstral can evaluate prompts, responses, prompt-response pairs, images, and images with accompanying text.

NVIDIA NeMo Guardrails is an open-source toolkit designed to enhance the safety, security, and compliance of large language model-based conversational applications. It enables developers to define, orchestrate, and enforce multiple AI guardrails, ensuring that generative AI interactions remain accurate, appropriate, and on-topic. The toolkit leverages Colang, a specialized language for designing flexible dialogue flows, and integrates seamlessly with popular AI development frameworks like LangChain and LlamaIndex. NeMo Guardrails offers features such as content safety, topic control, personal identifiable information detection, retrieval-augmented generation enforcement, and jailbreak prevention. Additionally, the recently introduced NeMo Guardrails microservice simplifies rail orchestration with API-based interaction and tools for enhanced guardrail management and maintenance.