feat: multi Agent OS server support in `AskUiControllerClient` by mlikasam-askui · Pull Request #276 · askui/python-sdk

mlikasam-askui · 2026-05-12T09:00:49Z

Summary

AskUiControllerClient can now manage multiple Agent OS target computers
(local and/or remote) at once. One is active at a time and receives the agent's
actions; callers can switch at runtime or scope a switch to a with block.

New API:

AgentOsTargetComputer (base), LocalAgentOsTargetComputer (replaces the old
AskUiControllerServer — owns the local controller subprocess; auto-detects
the Windows AskuiCoreService and switches to its port),
RemoteAgentOsTargetComputer (already-running remote controller, no process
management).
AgentOsTargetComputerManager — enforces invariants (at most one local,
unique session GUIDs / computer_ids / remote addresses) and owns the gRPC
connection lifecycle. Targets are addressed exclusively by computer_id
(single dict lookup; no secondary indices); the connection dict is keyed by
computer_id too.
On AskUiControllerClient and exposed through AgentOs /
ComputerAgentOsFacade: add_agent_os_target_computer,
add_remote_agent_os_target_computer, list_agent_os_target_computers,
get_current_computer_target_id (returns the computer_id string of the
active target), switch_agent_os_target_computer,
reset_agent_os_target_computers, temporary_select.
AndroidAgentOs / PpadbAgentOs get a sibling temporary_select(device_sn).
ComputerAgent auto-registers three new act() tools so the LLM can drive
multi-target flows: ComputerListAgentOsTargetComputersTool,
ComputerSwitchAgentOsTargetComputerTool,
ComputerGetCurrentComputerTargetIdTool.
LocalAgentOsTargetComputer and RemoteAgentOsTargetComputer are re-exported
from the top-level askui package.

New unit tests cover the target-computer classes, the manager, the multi-target
client, and the new computer tools. The e2e test was updated to the new
constructor.

Replace single-controller config with a TargetComputer / TargetComputerManager abstraction. AskUiControllerClient now manages a list of local and remote controller servers, opens a gRPC connection per target on connect(), and routes agent-os actions to a single active target. Add tools for the agent to list, switch, and inspect the active target computer.

… target session GUID

This reverts commit 7bd98be.

philipph-askui · 2026-05-14T08:44:45Z

 from .set_active_display_tool import ComputerSetActiveDisplayTool
+from .switch_agent_os_server_tool import ComputerSwitchAgentOsServerTool


shouldn't these 3 new tools also be in experimental?

I see them as part of the default tool list, similar to the Android agent.

philipph-askui · 2026-05-14T08:47:29Z

        cursor_position = self.agent_os.get_mouse_position()
-        return f"Mouse is at position ({cursor_position.x}, {cursor_position.y})."
+        return (
+            f"[Server with id '{server.computer_id}']: Mouse is at position "


I am not happy with the name "Server in general". So far, I was using the term "machine" when telling the agent that it is operating multiple devices.
The name of one machine could then be server. If we use the term server here, this will introduce ambiguity that might cause the currently working stuff to break

philipph-askui · 2026-05-14T08:47:44Z

-        return str(self.agent_os.get_system_info().model_dump_json())
+        server = self.agent_os.get_active_agent_os_server(report=False)
+        system_info_json = self.agent_os.get_system_info().model_dump_json()
+        return f"[Server with id '{server.computer_id}']: {system_info_json}"


same here (Server ->machine)

philipph-askui · 2026-05-14T08:48:47Z

+            agent_os=agent_os,
+        )
+
+    def __call__(self, computer_id: str) -> str:


how is this computer_id determined? By the user, e.g. in the init of the agent? Currently, they can set local_machine_name and remote_machine_name

It can be set by the user, or it defaults to the session ID, which is a UUID4.

philipph-askui · 2026-05-14T08:52:02Z

tbh: at the moment I really like the concept with 2 dedicated toolsets to operate 2 machines. Still, I see that this here will scale also to even more machines.

Have you checked if the agent understands that it can operate multiple machines and that it calls the switch tool before executing operations if needed?

Further: we need to automatically adjust the system capabilities and device information prompts if multiple agentOS are added.

Also: we should definitely update the docs with this PR to explain these changes here

programminx-askui · 2026-05-15T07:00:58Z

+        self, required_tags: list[str]
+    ) -> AgentOs | AndroidAgentOs:
+        """
+        Find the first registered agent OS whose tags are a superset of


Suggested change

Find the first registered agent OS whose tags are a superset of

Find the first registered AgentOS whose tags are a superset of

Please find all occurance. in Strings

programminx-askui · 2026-05-15T07:02:48Z

+
+    def temporary_select(self, device_sn: str) -> AbstractContextManager[Self]:


Suggested change

def temporary_select(self, device_sn: str) -> AbstractContextManager[Self]:

@abstractmethod

def temporary_select(self, device_sn: str) -> AbstractContextManager[Self]:

Ok.

Why is this function not implmented?

programminx-askui · 2026-05-15T07:06:14Z

        raise AndroidAgentOsError(msg)

+    @contextmanager
+    def temporary_select(self, device_sn: str) -> Iterator[Self]:


But in general. Why do we need the concept of temporary_select? How does the agent should call it?

temporary_select is not intended to be called directly by the agent. It is designed to allow users to execute commands on a specific device without needing to restore the previous device context afterward.

It can also be used in custom tools to restrict a tool to a specific device. For example, an enableWifiOnCarEmulator tool could internally use temporary_select, ensuring it only operates on the car emulator. This differs from a generic enableWifi tool, which works across all devices, where the agent is responsible for switching to the appropriate device before invoking the tool.

programminx-askui · 2026-05-15T07:07:09Z

+logger = logging.getLogger(__name__)
+
+
+def _generate_session_guid() -> str:


Do we have already such a function?

programminx-askui · 2026-05-15T08:10:48Z

+                return server
+        return None
+
+    def switch(self, computer_id: str) -> AgentOsServer:


We can switch to the same computer? Should this not failing?

programminx-askui · 2026-05-15T08:11:10Z

+        Raises:
+            KeyError: If no server with the given computer id is registered.
+        """
+        server = self.get(computer_id)


Please wrap a proper AgentOSServerManager Error around.

programminx-askui · 2026-05-15T08:15:01Z

+    def agent_os_server_manager(self) -> AgentOsServerManager:
+        """The underlying Agent-OS-server manager."""
+        return self._manager


Why do we have the server_manager inside the client?

THis should not be here

programminx-askui · 2026-05-15T08:17:15Z

        self.is_cacheable = True

    def __call__(self) -> str:
+        server = self.agent_os.get_active_agent_os_server(report=False)


Intresting.

programminx-askui · 2026-05-15T08:19:21Z

+        assert _generate_session_guid() != _generate_session_guid()
+
+
+class TestReplacePort:


…skui-controller-multi-target

mlikasam-askui and others added 13 commits May 8, 2026 12:17

add reporter messgaes

768653c

fix logger

01e8f4f

refactor: drop tags from target computers and prefix tool output with…

cfb84f6

… target session GUID

fixed reporter

de55cf2

add temporary_select target

c7f9631

add device_id

aa0aba5

improve doc string

78523b4

improve error message

4ca7f80

add unit tests

9379663

improve tool results

e40c02c

fix lint problems

3493c4a

improve-doc-string

9a475c5

mlikasam-askui mentioned this pull request May 12, 2026

feat: multi Agent OS server support in AskUiControllerClient #275

Closed

mlikasam-askui requested review from philipph-askui and programminx-askui May 12, 2026 09:01

mlikasam-askui self-assigned this May 12, 2026

mlikasam-askui and others added 2 commits May 12, 2026 11:47

feat: capture unhandled errors in reporter

7bd98be

Revert "feat: capture unhandled errors in reporter"

24de514

This reverts commit 7bd98be.

philipph-askui reviewed May 14, 2026

View reviewed changes

philipph-askui requested changes May 14, 2026

View reviewed changes

mlikasam-askui requested a review from philipph-askui May 15, 2026 06:55

programminx-askui requested changes May 15, 2026

View reviewed changes

mlikasam-askui requested a review from programminx-askui May 19, 2026 09:54

implement review remarks

55cdac0

mlikasam-askui force-pushed the SOLENG-360-refactor/askui-controller-multi-target branch from c20d744 to 55cdac0 Compare May 19, 2026 10:08

Merge remote-tracking branch 'origin/main' into SOLENG-360-refactor/a…

40873be

…skui-controller-multi-target

		from .set_active_display_tool import ComputerSetActiveDisplayTool
		from .switch_agent_os_server_tool import ComputerSwitchAgentOsServerTool

	Find the first registered agent OS whose tags are a superset of
	Find the first registered AgentOS whose tags are a superset of


		def temporary_select(self, device_sn: str) -> AbstractContextManager[Self]:

		logger = logging.getLogger(__name__)


		def _generate_session_guid() -> str:

		assert _generate_session_guid() != _generate_session_guid()


		class TestReplacePort:

Conversation

mlikasam-askui commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philipph-askui commented May 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mlikasam-askui commented May 12, 2026 •

edited

Loading