Codestin Search App

History

193 lines (178 loc) · 10.8 KB

Raw

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

<!DOCTYPE html>

<html>

<head>

<title>WorldGUI</title>

<meta

name="description"

content="WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point"

<meta

name="viewport"

content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no"

<link

rel="stylesheet"

href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.2.0/css/all.min.css"

integrity="..."

crossorigin="anonymous"

<script

async

src="https://www.googletagmanager.com/gtag/js?id=G-H9XFCMDPNS"

></script>

window.dataLayer = window.dataLayer || [];

function gtag() {

dataLayer.push(arguments);

}

gtag("js", new Date());

gtag("config", "G-H9XFCMDPNS");

</script>

</head>

<body>

<section style="

background: linear-gradient(90deg, #00C9FF 0%, #92FE9D 100%);

background-blend-mode: multiply;">

<div

class="content-wrapper title-wrapper"

style="flex-direction: column;"

<div

style="

display: flex;

flex-direction: row;

align-items: center;

padding-bottom: 15px;

<h1 style="font-size: 60px; padding-top: 0.4em; color: #2F4F4F;">WorldGUI</h1>

<img

src="assets/desktop.png"

style="height: 60px; padding-top: 0em; padding-left: 1em; margin-bottom:-20px;"

</div>

<h2 style="color: #2F4F4F;">An Interactive Benchmark for Desktop GUI Automation from Any Starting Point</h2>

Henry Hengyuan Zhao, Kaiming Yang, Wendi Yu, Difei Gao, Mike Zheng Shou

</p>

National University of Singapore

</p>

<!-- <a href="index.html">

<button

class="outline multimodal"

style="flex-direction: row; display: flex; justify-content: center; align-items: center;">

<img

src="img/swellama.png"

style="height: 1.3em; margin-right: 0.4em; margin-bottom: 0.3em;" />

Home 

</button>

</a> -->

<i class="fa fa-paperclip"></i> Paper 

</button>

</a>

<i class="fa-brands fa-github"></i> Code 

</button>

</a>

</div>

</section>

<h2 class="text-title" style="margin-bottom:0.5em">What's new with WorldGUI Benchmark?</h2>

<p class="text-content">TL;DR: WorldGUI extends the evaluation of GUIs from a static to a dynamic testing process, which is more relevant for reflecting the complex and dynamic nature of GUI environments.</p>

<p class="text-content">WorldGUI is an early work to stimulate <i>dynamism</i> in the real user-computer scenarios.

As illustrated in above figure, most GUI benchmarks focus on initial and final states, measuring success rates but overlooking the changing initial conditions present in real GUI scenarios.

These benchmarks often <b>ignore</b> situations where: <br>(1) The software interface is not in its default state. <br> (2) The agent might get user queries at any time. <br>(3) Differences in agent robustness, where agents with the same low success rate (e.g. 2%) may vary

in their ability to self-verify or self-correct, but these abilities are not measured in a <i>static</i> setting.</p>

<h2 class="text-title" style="margin-bottom:0.5em">Overview</h2>

Software taxonomy of WorldGUI and the performance comparison of GUI agents. The left shows 5 main groups and 10 software in our WorldGUI. The right shows that WorldGUI-Agent surpasses previous SOTA GUI agents significantly.

</p class="text-content">

<b>WorldGUI</b>: The left shows that for each task, WorldGUI provides a user query, instructional video, and pre-actions. The pre-actions lead to different initial states. The key characteristic of our WorldGUI is the various initial states of the same task to stimulate the real-world testing process. The right shows the software included in our benchmark and the interactions about testing the agents in our GUI environment.

</p class="text-content">

<h2 class="text-title" style="margin-bottom:0.5em">WorldGUI-Agent Framework</h2>

<b>WorldGUI-Agent</b> includes five proposed components: Planner, Planner-Critic, Step-Check, Actor, and Actor-Critic. The Planner module receives the user query and an instructional video as input and generates an initial plan for the Planner-Critic process. This plan is then refined and executed step by step. Before each step is passed to the Actor module, it undergoes a Step-Check. After the Actor produces an action, the Actor-Critic module iteratively verifies the completion of the action and makes corrections if needed.

</p class="text-content">

<h2 class="text-title">Benchmark Comparison</h2>

Table 1: WorldGUI is a unique benchmark that has the various states for each task to stimulate the real-world agent-computer interactions.

</p>

<h2 class="text-title">Data Statistic</h2>

<p class="text-content" style="text-align: center;">Table 2: This table shows all tasks, task activities, and project file of the desktop applications used in WorldGUI.</p>

<h2 class="text-title">An Successful Execution Example</h2>

<h2 class="text-title">An Example of Augmented Data Construction</h2>

<h2 class="text-title">Visualization of Parser Results</h2>

<h2 class="text-title">An Example of Planner-Ciritc</h2>

<h2 class="text-title">An Example of Step-Check</h2>

<h2 class="text-title">An Example of Actor-Ciritc</h2>

<h2 class="text-title">Erorr Cases Visualization</h2>

<h2 class="text-title" style="margin-bottom:0.5em">Citation</h2>

<pre id="citation" style="border-color: #2F4F4F;"><code>@misc{zhao2025worldguiinteractivebenchmarkdesktop,

title={WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point},

author={Henry Hengyuan Zhao and Kaiming Yang and Wendi Yu and Difei Gao and Mike Zheng Shou},

year={2025},

eprint={2502.08047},

archivePrefix={arXiv},

primaryClass={cs.AI},

url={https://arxiv.org/abs/2502.08047},

}</code></pre>

<b>Acknowledge:</b> Thanks to Carlos & John for this webpage template.

Also thanks to the SWE-bench team and their benchmark <a href="https://www.swebench.com/multimodal.html">https://www.swebench.com/multimodal.html</a>.

</p>

<p><b>Template Usage:</b> If you would like to use this website template for your own leaderboard, please <span style="color:brown">send Carlos & John an email requesting permission.</span> If granted, please make sure to acknowledge the SWE-bench team and link to this leaderboard on the home page of the website.

</p>

<!-- <p class="text-content">

Correspondence to: <a href="hen">[email protected]</a>,

</p> -->

</a>

</a>

</div>

</section>

</div>

</body>

</html>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

FilesExpand file tree

index.html

Latest commit

History

index.html

File metadata and controls