Prompt management systems compared

At Nearform, we work extensively with large language models (LLMs) and AI agents, and understand that the quality of the prompts we feed into these systems have a major impact on their performance.

As our projects grew, managing prompts became more difficult - prompts were hardcoded into application code, and we often saw similar prompts spread across different codebases. To tackle these issues, we started using prompt management systems, and have seen significant improvements in efficiency and collaboration.

For example:

• Prompts are now managed independently of application code, allowing teams to update and refine them quickly without redeploying the entire application. This allowed teams to iterate on prompts much faster, eliminating bottlenecks.
• Shared prompt registries make it easy for teams across the organization to collaborate on and reuse prompts, such as those for:
  • Providing consistent context about the project or company
  • Defining clear output indicators for desired responses

By pooling efforts together, we reduced a lot of rework.

This article will provide an overview of how we evaluated the various prompt management systems on the market, which ones we recommend, and why.

When evaluating prompt management systems, we focused on how well they implemented the following features:

• Prompt versioning & registry - the ability to track the history of a prompt through different iterations
• Team collaboration - the ability for team members to share prompts with others
• Prompt composability - the ability to compose prompts by combining reusable snippets
• Prompt evaluation - the ability to evaluate prompts based on how well it guides the system to produce the desired response

We also factored in the size and level of involvement of the community, whether it is free and open-source, and whether using it will lead to vendor lock-in.

More on Prompt Evaluation

Prompt management systems enable engineers to iterate on prompts to make them better over time. What “better” means depends on how you evaluate the output. Apart from manual human review, there are a host of prompt and LLM evaluation metrics and methods that enable automatic evaluation. For text summarization, you may use a metric like BERTScore; for translation, perhaps COMET-QE. But the one that often yields the best result and is much more flexible is the LLM-as-a-Judge method, where we prompt another LLM to act as a judge and evaluate the output.

When evaluating each system’s features, we looked at:

• Manual review support - how easy is it to annotate and review results?
• Automated evaluation - does it support metrics (e.g. BERTScore, COMET-QE) or LLM-as-a-Judge methods?
• Integration options - can it connect with custom evaluators and/or external evaluation frameworks?

Through a process of elimination, we whittled down the list to:

• Langfuse - an open-source LLM engineering platform that provides metrics, observability, prompt management, and evaluation features. It is backed by Y Combinator.
• MLflow - originated as a platform for managing machine learning model training, but has recently introduced GenAI-oriented features like prompt management and evaluations. It is a Linux Foundation project, and its main contributor is Databricks.

In the next few sections, we will dive deeper into the features of Langfuse and MLflow. Towards the end of the article, we will also briefly go through the alternatives and explain why we did not adopt them.

Langfuse and MLflow Prompt Registry both implement automatic versioning, assigning a new version to a prompt each time it’s updated. Both also allow you to manually assign additional tags (called labels in Langfuse, and aliases in MLflow) to organize prompts based on your own versioning scheme.

Langfuse and MLflow both provide interfaces at different layers of abstractions to manage prompts and their versions.

Langfuse provides an HTTP API, complete with an OpenAPI specification and Postman collection. On top of the API are the official Python and TypeScript SDKs, which provide additional features such as caching.

MLflow also provides an API and a set of SDKs, but its setup is a bit more nuanced. In the open-source version, there are no HTTP API endpoints for managing prompts. Instead, the Python SDK stores prompts as a special type of machine learning model, effectively hijacking the Model Registry API to manage prompts.

Both Langfuse and MLflow provide a web-based user interface for managing prompts and their versions. Langfuse’s UI is clean and slick, and you can play around with it via their interactive demo. MLflow’s UI has a more dated design, and you can see some screenshots of the UI from its documentation.

Langfuse supports authentication via email/password, social logins, and single sign-on (SSO) using OpenID Connect (OIDC). Once authenticated, administrators can give permissions to users via a role-based access control system.

Langfuse offers 5 predefined roles - Owner, Admin, Member, Viewer, and None - each defined as a list of permission scopes (e.g. prompts:read). Administrators can assign users roles at the organization or project level (a project is a subdivision of an organization; assigning roles per-project is a feature that’s only available in Enterprise Edition).

MLflow has added support for HTTP basic authentication and authorization since v2.5.0, but as of v3.4.0, is still marked as experimental. In terms of authorization, it supports granting permissions on two resource types - experiments and registered models - and four levels of permissions - READ, EDIT, MANAGE, and NO_PERMISSIONS. As we discussed earlier, prompts are stored as models in the open-source version, and so it is possible to manage permissions for prompts by setting permissions on models. For the Databricks-managed version, you’d configure permissions using either MLflow model access control lists, or Unity Catalog privileges if Unity Catalog is enabled.

Both Langfuse and MLflow allow you to define prompt templates with {{ placeholders }} that get populated with data at runtime. In addition, Langfuse also supports referencing other prompts inside a prompt using a special syntax: (@@@langfusePrompt:name=PromptName|label=production@@@).

In Langfuse, evaluation works by feeding a dataset or live data into an LLM and applying evaluation methods to the output to generate a Score data object representing the performance of the LLM and/or the prompt. Evaluation methods include LLM-as-a-Judge, human annotations, and custom scoring via API/SDK.

For LLM-as-a-Judge, Langfuse provides a large list of managed evaluators that you can use out-of-the-box. For human annotation, it has a queue system where you can assign users to work on specific queues. Evaluators can add annotations at the trace level (representing one prompt-response cycle), a session (a group of traces representing the entire user interaction), or observations (individual steps within a trace). Lastly, the custom scoring feature is a flexible way for you to integrate with your own evaluator implementation (e.g. if you prefer evaluating using more traditional metrics) and/or to any external evaluation frameworks.

MLflow works similarly, but many evaluation features are only available in the Databricks-managed version and are missing from the open-source release. For LLM-as-a-Judge, it provides a list of predefined scorers, although that list is shorter than Langfuse’s and some are only available on Databricks managed MLflow. MLflow supports basic human evaluation, but nothing more. MLflow also supports custom code-based scorers, similar to Langfuse’s custom scoring.

One feature that MLflow has over Langfuse is its Agent-as-a-Judge scorer, which is similar to LLM-as-a-Judge, but the judge acts as an agent and is able to call tools via the Model Context Protocol (MCP) and potentially provide more accurate evaluations.

Community

Both Langfuse and MLflow have a large community and commercial backers behind them:

Open-source model

Both Langfuse and MLflow are open-core. But while Langfuse’s OSS version contains most of the important features, MLflow kept many of its important features behind the managed version.

Self-Hosting

Langfuse makes it easy to self-host. There is a decent self-hosting guide, and there are official Terraform modules (e.g. langfuse-terraform-aws) and community-maintained Helm charts.

Conversely, on MLflow, there is a tutorial on running a MLflow Tracking Server, which the GenAI features are built on top of, but no dedicated self-hosting guide nor official Terraform modules or Helm charts.

Although we recommend Langfuse and MLflow, it’s useful to know why we opted not to adopt the alternatives. So here are a brief summary of those that didn’t make the list:

• Helicone - Helicone is primarily an AI gateway that provides observability, analytics, cost-saving features (caching and automatic model selection), while also providing prompt management features that can be used independently of the gateway (via its TypeScript and Python SDK). Prompt management is not its focus so, understandably, it is not as full-featured as dedicated prompt management systems; for instance, it lacks the ability to run evaluations (but does provide a way to centralize results from external evaluation frameworks).
• Agenta - Agenta markets itself as an all-in-one LLMOps platform that provides prompt management, LLM evaluations, and observability. In terms of prompt management, prompts are not first-class citizens on the platform; for example, you cannot version individual prompts or snippets, but instead only variants of an application. Agenta also uses an open-core model, and the lack of evaluation features in the open-source version not suitable for us.
• Prompt flow - Prompt flow is a suite of tools for developing LLM-based AI applications. However, similar to Agenta, you cannot version prompts, only entire LLM applications. It also suffers from a lack of activity (last release was 10 months ago, in January 2025) and care (documentation is riddled with errors). Lastly, it currently only supports OpenAI models and models on Azure AI Foundry, leading to vendor lock-in.
• Arize Phoenix - an open-source AI observability platform that includes prompt management features such as prompt versioning, prompt templating, prompt registry, user management, and evaluations. You are also able to self-host it. However, their client libraries still at an early stage of development (e.g. no support for caching).
• LangSmith - it is closed-source and only available as a managed service, leading to vendor lock-in.
• Braintrust - a development platform for LLM-powered applications, but is closed-source and does not fully support self-hosting.
• PromptLayer - has most of the desired features, but self-hosting requires an Enterprise license.
• Lilypad - a very new development platform for LLM-powered applications, has fantastic documentation, but lacks many features and relies on a single main contributor.

Prompt management systems are not without their downsides. Your application now depends on an external networked service, which can become unreachable and adds complexity to your application.

Service interruptions can be somewhat mitigated with caching at the SDK layer (using a cached version of the prompt if the service is unreachable) and by providing fallbacks. With these mitigations in place, the benefits of adopting prompt management systems far outweighed the negatives - we were able to iterate quicker on prompts and increased developer productivity.

At the time of this writing (October 2025), Langfuse stands heads and shoulders above everyone else in the prompt management space. But teams that only need basic prompt management features may also want to consider MLflow if they are using it already, in order to avoid having to operate an additional platform.