Skip to content

Improve trsh attempts reporting #871

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Lilachn91 wants to merge 1 commit into
base: main
Choose a base branch
from
+40 −4
Open

Improve trsh attempts reporting #871

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions arc/job/trsh.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1137,10 +1137,22 @@ def trsh_ess_job(label: str,
couldnt_trsh = True

if couldnt_trsh:
logger.error(f'Could not troubleshoot geometry optimization for {label}! '
f'Tried troubleshooting with the following methods: {ess_trsh_methods}')
# Count and remove 'trsh_attempt' entries for cleaner reporting
trsh_attempt_count = ess_trsh_methods.count('trsh_attempt')
filtered_methods = [method for method in ess_trsh_methods if method != 'trsh_attempt']

# Build the message with the count and filtered methods
if trsh_attempt_count > 0 and filtered_methods:
message = f'Tried troubleshooting {trsh_attempt_count} times, with the following methods: {filtered_methods}'
elif trsh_attempt_count > 0:
Copy link
Copy Markdown
Member

@calvinp0 calvinp0 Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a time when there would be no methods? I mean if it did attempt to troubleshoot x number of times, then it would have used a method, I think?

Copy link
Copy Markdown
Member

@calvinp0 calvinp0 Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, when there is no methods applied, so no troubleshoot could be done (an edge case I guess?) Should then the message be something like "No applicable troubleshooting methods found".

message = f'Tried troubleshooting {trsh_attempt_count} times'
Comment on lines +1143 to +1148
Copy link

Copilot AI Apr 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The attempt-count message always uses "times" (e.g., "Tried troubleshooting 1 times"), which reads incorrectly for a single attempt. Consider pluralizing based on the count ("1 time" vs "N times") so the error/output logs are clearer and more polished.

Suggested change
# Build the message with the count and filtered methods
if trsh_attempt_count > 0 and filtered_methods:
message = f'Tried troubleshooting {trsh_attempt_count} times, with the following methods: {filtered_methods}'
elif trsh_attempt_count > 0:
message = f'Tried troubleshooting {trsh_attempt_count} times'
attempt_label = 'time' if trsh_attempt_count == 1 else 'times'
# Build the message with the count and filtered methods
if trsh_attempt_count > 0 and filtered_methods:
message = f'Tried troubleshooting {trsh_attempt_count} {attempt_label}, with the following methods: {filtered_methods}'
elif trsh_attempt_count > 0:
message = f'Tried troubleshooting {trsh_attempt_count} {attempt_label}'

Copilot uses AI. Check for mistakes.
else:
message = f'Tried troubleshooting with the following methods: {filtered_methods}'

logger.error(f'Could not troubleshoot {job_type} for {label}! '
f'{message}')
output_errors.append(f'Error: Could not troubleshoot {job_type} for {label}! '
f'Tried troubleshooting with the following methods: {ess_trsh_methods}; ')
f'{message}; ')
return (output_errors,
ess_trsh_methods,
remove_checkfile,
Expand Down
21 changes: 21 additions & 0 deletions arc/job/trsh_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -445,6 +445,27 @@ def test_trsh_ess_job(self):
output_errors,
)

# Gaussian: test 6b - verify troubleshoot attempts counting
job_status = {'keywords': ['MaxOptCycles', 'GL9999']}
ess_trsh_methods = ['trsh_attempt',
'int=(Acc2E=14)', 'opt=(maxcycle=200)',
'trsh_attempt', 'opt=(RFO)',
'trsh_attempt', 'opt=(GDIIS)',
'trsh_attempt', 'opt=(GEDIIS)',
'trsh_attempt']
output_errors, ess_trsh_methods, remove_checkfile, level_of_theory, software, job_type, fine, trsh_keyword, \
memory, shift, cpu_cores, couldnt_trsh = trsh.trsh_ess_job(label, level_of_theory, server, job_status,
job_type, software, fine, memory_gb,
num_heavy_atoms, cpu_cores, ess_trsh_methods)
self.assertTrue(couldnt_trsh)
self.assertIn('Tried troubleshooting 5 times, with the following methods:', output_errors[-1])
self.assertNotIn('trsh_attempt', output_errors[-1])
self.assertIn("opt=(maxcycle=200)", output_errors[-1])
self.assertIn("opt=(RFO)", output_errors[-1])
self.assertIn("opt=(GDIIS)", output_errors[-1])
self.assertIn("opt=(GEDIIS)", output_errors[-1])
self.assertIn('all_attempted', output_errors[-1])

# Gaussian: test 7
job_status = {'keywords': ['MaxOptCycles', 'GL9999','SCF']}
ess_trsh_methods = ['int=(Acc2E=14)']
Expand Down
5 changes: 4 additions & 1 deletion arc/scheduler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3554,7 +3554,8 @@ def troubleshoot_ess(self,

level_of_theory = Level(repr=level_of_theory)
logger.info('\n')
warning_message = f'Troubleshooting {label} job {job.job_name} which failed'
# log job failure information before troubleshooting
warning_message = f'{label} Job {job.job_name} failed'
if job.job_status[1]["status"] and job.job_status[1]["status"] != 'done':
warning_message += f' with status: "{job.job_status[1]["status"]},"'
if job.job_status[1]["keywords"]:
Expand Down Expand Up @@ -3593,6 +3594,8 @@ def troubleshoot_ess(self,
f'Reached max troubleshooting attempts ({max_ess_trsh}).')
self.output[label]['errors'] += f'Error: ESS troubleshooting attempts exhausted for {label} {job.job_type}; '
return
logger.warning(f'Troubleshooting {label} job {job.job_name} '
f'(attempt number {trsh_attempts + 1}).')
Copy link
Copy Markdown
Member

@calvinp0 calvinp0 Apr 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is more a readability suggestion but we could do something like this:

next_attempt = trsh_attempts + 1
logger.warning(
    f'Troubleshooting {label} job {job.job_name} '
    f'(attempt number {next_attempt}).'
)
job.ess_trsh_methods.append('trsh_attempt')

That way if someone were to look at this code in the future, it would be quick to see that its the next attempt

job.ess_trsh_methods.append('trsh_attempt')

# Determine if the species is a hydrogen atom (or its isotope).
Expand Down
Loading