Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tok/sec calc in demo.py should not include TTFT #88

Closed
mattjcly opened this issue Feb 2, 2025 · 0 comments · Fixed by #92
Closed

Tok/sec calc in demo.py should not include TTFT #88

mattjcly opened this issue Feb 2, 2025 · 0 comments · Fixed by #92
Labels
bug Something isn't working good first issue Good for newcomers

Comments

@mattjcly
Copy link
Member

mattjcly commented Feb 2, 2025

mlx-engine/demo.py

Lines 114 to 122 in cb1b880

def print_stats(self):
"""Print generation statistics."""
end_time = time.time()
total_time = end_time - self.start_time
print(f"\n\nGeneration stats:")
print(f" - Time to first token: {self.first_token_time - self.start_time:.2f}s")
print(f" - Total tokens generated: {self.total_tokens}")
print(f" - Total time: {total_time:.2f}s")
print(f" - Tokens per second: {self.total_tokens / total_time:.2f}")

Tokens per second should be calculated by dividing by total_time - time_to_first_token instead of just total_time

@mattjcly mattjcly added bug Something isn't working good first issue Good for newcomers labels Feb 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant