Interface Graphs Infrastructure
How they are built and used as the user interface
The Relayer App is powered by the agent, which serves as an interface that manages the workspace and user session via interface graphs. Interface graphs are what the user sees in the app. Interactions such as selecting an action, opening a node, or speaking to the agent are processed in real time.
Local MCP servers write to the conversation and workspace graph, and are a critical improvement for the graph creation. They allow the interface graph to be streamed to the user as the tool is repeatedly called during reasoning. The MCP server also embeds the strict node and turn logic to ensure the agent adheres to the instructions and builds graphs that are usable in the frontend. There is a fine balance in these validation rules, but we currently optimize for coherent nodes and infinite interactivity. At any given point in the app, there should be places to go and things to do.
As these graphs are built, a vector index is constructed using the FAISS library for simple retrieval and deduplication. This allows the agent to quickly search pre-computed reasoning and also provide ground truth for the workspace or conversation state. When the user interacts with a node, the agent can see related nodes and consider those in context when understanding the action. This allows the agent to connect the dots between sequential actions and build a coherent conversation that moves forward while remembering short and long-term context.
For a responsive user experience, actions are processed quickly, and heavy reasoning tasks are handled by a pool of workers, allowing us to balance latency and compute. Depending on the state of the session, we allocate more workers to the turn task queue as that is relevant to the users’ current queries. When the user is not connected and there are no active threads, the workspace graph building will take priority. Within a single thread, we can also process one set of actions at a time, avoiding race conditions when writing the graph. As a result, the task queue will accumulate actions per thread, allowing us to reduce the number of overall agent calls and reach the desired state sooner.
The node expansion queue is a recursive process that starts at the workspace's root. Each agent's turn for node expansion outputs a list of (node label to expand, priority score). This feeds back into the expansion queue, so the workspace graph is continuously built by order of importance. The agent uses signals such as node depth, user interaction history, and current conversation topics to determine which nodes to expand and to predict what information the user may want to dive deeper into.
The agent-as-interface approach is currently being applied for development, using the codebase as the workspace. Still, the infrastructure is generic and can be applied across different knowledge-work domains. Graphs, nodes, and state machines are base objects, and the future direction is to expand the ways the agent can represent information and complete tasks, making this technology more effective across a range of scenarios.