Vyokky/dev Agent and automator modularization + Learning for demonstration #50

vyokky · 2024-04-02T13:05:03Z

Extract agent module for HostAgent and AppAgent, intergrate into main flow.
Build automator module for AppAgent, for UI controls and API.
Learning from human demonstration (Yun Hao)

Enable human demonstration RAG

kangyu · 2024-04-03T16:11:28Z

record_processor/parser/psr_record_parser.py

+        """
+
+        user_action_data = re.search(
+            r'<UserActionData>(.*?)</UserActionData>', content, re.DOTALL)


There are many hard-code implementation like this. Need some regression test to secure, in case the PSR changes its format in future release.

Agree on the regression test. Will discuss offline about the implementation.

kangyu · 2024-04-03T16:14:05Z

record_processor/README.md

+UFO can learn from user-provided demonstrations for specific requests and use them as references in the future when encountering similar tasks. Providing clear demonstrations along with precise requests can significantly enhance UFO's performance.
+
+## How to Enable and Config this Function ❓
+You can enable this function by setting the following configuration:


Need to mention which configuration file as we have two in UFO to set these configs.

Good idea, will update.

kangyu · 2024-04-03T16:16:39Z

record_processor/parser/psr_record_parser.py

+                screenshot = self.get_screenshot(screenshot_file_name)
+                step_key = f"step_{int(action_number) - 1}"
+
+                step = DemonstrationStep(


Some of the fields may not be extracted correctly in some cases (e.g., PSR changes it format). May need sanity check on the values.

kangyu · 2024-04-03T16:31:33Z

record_processor/parser/psr_record_parser.py

+
+        return record
+
+    def find_boundary(self) -> str:


These are private methods, could use private function definition.

Suggested change

def find_boundary(self) -> str:

def __find_boundary(self) -> str:

Yes, will update

kangyu · 2024-04-03T16:34:43Z

record_processor/summarizer/summarizer.py

+            summary = dict()
+            summary["example"] = {}
+            for key in ["Observation", "Thought", "ControlLabel", "ControlText", "Function", "Args", "Status", "Plan", "Comment"]:
+                summary["example"][key] = response_json.get(key, "")


Why we use empty string if the key is not in the set? May add some comments on this.

kangyu · 2024-04-04T09:48:27Z

ufo/module/flow.py

        self.logger = self.initialize_logger(self.log_path, "response.log")
        self.request_logger = self.initialize_logger(self.log_path, "request.log")

-        self.app_selection_prompter = ApplicationAgentPrompter(configs["APP_AGENT"]["VISUAL_MODE"], configs["APP_SELECTION_PROMPT"], configs["APP_SELECTION_EXAMPLE_PROMPT"], configs["API_PROMPT"])
-        self.act_selection_prompter = ActionAgentPrompter(configs["ACTION_AGENT"]["VISUAL_MODE"], configs["ACTION_SELECTION_PROMPT"], configs["ACTION_SELECTION_EXAMPLE_PROMPT"], configs["API_PROMPT"])
+        self.HostAgent = HostAgent("HostAgent", configs["HOST_AGENT"]["VISUAL_MODE"], configs["HOSTAGENT_PROMPT"], configs["HOSTAGENT_EXAMPLE_PROMPT"], configs["API_PROMPT"])


We may have some way to protect the value errors in config class. In case some config items are missing.

kangyu · 2024-04-04T09:53:29Z

ufo/module/flow.py

@@ -81,113 +74,118 @@ def process_application_selection(self):
        """

        # Code for selecting an action


This function is too long to read. May split it into several sub functions, to reveal the logic steps.

Agree. Will update the func and break it into different sub-steps.

ufo/module/flow.py

kangyu · 2024-04-04T10:07:56Z

ufo/module/flow.py

+
+            if self.safe_guard(action, control_text):
+                # Execute the action
+                results = ui_controller.execution(function_call, args)


UI controller to execute function call is wired. These function calls are actually operations instead of API calls, right? What about use different name here?

kangyu · 2024-04-04T10:10:35Z

ufo/module/flow.py

@@ -1,31 +1,28 @@
 # Copyright (c) Microsoft Corporation.


In general, I think this file is too long. Some functions could be split and put into different files/classes. Otherwise, it is not easy to read and follow.

I agree. Perhaps will break it down for next update.

yunhao0204 and others added 30 commits March 26, 2024 14:05

Add record processor for the user demonstration learning

92f354e

Polish code and enable UFO Rag from user demostration

398a778

Add README and related sample files

73e91cf

basic agent classes

b79e5f2

basic classes

1c5fa1d

import refine

db478e9

agent framework

db0c127

agent framework

a41a171

log and memory management

d29bf1d

plugin

23e42c0

plugin

8a8536f

Enable record_processor return multiple plans to let user choose

1c7ad6f

plugin basic

0ff8289

plugin basic

8df4178

refinement

d2ca2c4

refinement

fc3fb33

refinement

8483a1b

app puppeteer

40cb6d9

app puppeteer

8c8f81b

sorted imported

7531be5

name change

923ee6c

name change

833b6e2

Fix the json parse issue and update README

3b87cca

readme for offline RAG

654da0f

readme for offline RAG

d083ebf

puppeteer init

a530d84

puppeteer init

29c2e22

Merge branch 'vyokky/dev' into demonstration

4fb0937

Merge pull request #49 from yunhao0204/demonstration

c11b097

Enable human demonstration RAG

agent name change

658a86d

vyokky added 8 commits April 2, 2024 20:53

config fixed

eb96de9

config fixed

9ca9b2b

error trace

3c108b8

rm redundant

f89be86

merge

5adb714

bug fix

25d5923

bug fix

733e282

color change

1149b02

vyokky requested a review from kangyu April 2, 2024 13:25

vyokky assigned kangyu and vyokky Apr 2, 2024

vyokky added 7 commits April 2, 2024 21:29

readme

988f110

rm redundant

86f4213

rm old prompt

55dfb83

sort imports

591f945

check com exist

8b8f464

comment and docstring

c5da96f

comment and docstring

e7f734a

kangyu reviewed Apr 5, 2024

View reviewed changes

vyokky added 4 commits April 7, 2024 22:35

fix comment

7f5c58c

fix comment

123fa88

fix agent register

718e74b

memory fix

adc8b13

vyokky merged commit 1f6312e into pre-release Apr 8, 2024
1 check passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vyokky/dev Agent and automator modularization + Learning for demonstration #50

Vyokky/dev Agent and automator modularization + Learning for demonstration #50

vyokky commented Apr 2, 2024

kangyu Apr 3, 2024

yunhao0204 Apr 6, 2024

kangyu Apr 3, 2024

yunhao0204 Apr 6, 2024

kangyu Apr 3, 2024

kangyu Apr 3, 2024

yunhao0204 Apr 6, 2024

kangyu Apr 3, 2024

kangyu Apr 4, 2024

kangyu Apr 4, 2024

vyokky Apr 7, 2024

kangyu Apr 4, 2024

kangyu Apr 4, 2024

vyokky Apr 7, 2024

	def find_boundary(self) -> str:
	def __find_boundary(self) -> str:

		@@ -81,113 +74,118 @@ def process_application_selection(self):
		"""

		# Code for selecting an action

Vyokky/dev Agent and automator modularization + Learning for demonstration #50

Vyokky/dev Agent and automator modularization + Learning for demonstration #50

Conversation

vyokky commented Apr 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment