I am sorry for asking a lot of questions but this was something that i encountered today while running echo echo meow | podrun -i command. So i created a tpu pod for a tpuv4-32, so the podrun command should print meow 4 times but what happened was it printed it 3 times and through an AuthenticationException('Authentication failed.')

output for the command:
echo echo meow | podrun -i

meow
meow
meow
Traceback (most recent call last):
  File "/home/my_username/.local/bin/podrun", line 8, in <module>
    sys.exit(main())
  File "/home/my_username/.local/lib/python3.10/site-packages/tpux/podrun.py", line 25, in main
    run_command_on_all_hosts(command, include_local=args.include_local)
  File "/home/my_username/.local/lib/python3.10/site-packages/tpux/utils.py", line 44, in run_command_on_all_hosts
    group.run(command)
  File "/home/my_username/.local/lib/python3.10/site-packages/fabric/group.py", line 116, in run
    return self._do("run", *args, **kwargs)
  File "/home/my_username/.local/lib/python3.10/site-packages/fabric/group.py", line 282, in _do
    raise GroupException(results)
fabric.exceptions.GroupException: {<Connection host=IP_1>: <Result cmd='echo meow\n' exited=0>,<Connection host=IP_2>: <Result cmd='echo meow\n' exited=0>,<Connection host=IP_3>: <Result cmd='echo meow\n' exited=0>,Connection host=IP_4"**This wasn't same as what i had entered during the setup process of the TPU pod**">: AuthenticationException('Authentication failed.')}

can you please help me understand the reason for this?

Thanks in advance for being so patient and answering my queries :)

Can we use tpux to run pytorch/xla code on tpu pods? #5

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions