Is Linguist used for *syntax highlighting* or just language detection?

[verhovsky]

I am trying to report this highlighting issue

https://github.com/verhovsky/curlconverter/blob/869c6d0229a9004bdca8e44d1ef3c3227e785093/extract_curl_args.py#L201

and I've wasted 30 minutes trying to figure out where to report it. I googled "what does github use for syntax highlighting" and the first result says that according to https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-repository-languages it's Linguist. However if you read carefully that page only says (emphasis mine)

GitHub uses the open source Linguist library to determine file languages for syntax highlighting and repository statistics

depending on how you read it, it doesn't say it uses Linguist to do the actual highlighting after determining the language and some people are pointing that out in the comments on Stack Overflow as well. I saw in the repo that it has the latest version of MagicPython as a submodule so I filed an issue there

MagicStack/MagicPython#239

but then I noticed this other issue MagicStack/MagicPython#225 where someone had an issue with Github's syntax highlighting but MagicPython highlights it correctly and a MagicPython contributor is saying that it doesn't look like Github is using it for the syntax highlighting. I opened the snippet in VS Code (which presumably uses MagicPython, but I don't know) and it was highlighted correctly.

[Nixinova]

GitHub Linguist allows for syntax highlighting but the highlighting is done through external modules, not Linguist itself. You can see what grammar is used at https://github.com/github/linguist/tree/master/vendor#grammar-index and issues are to be reported in the upstream grammar repo instead of here. So yes that repo is the correct place to go.

As to the issue 255 linked, @lildude may be of help there

[lildude]

Linguist does not do the syntax highlighting, only language detection.

Syntax highlighting is performed by a service that uses tree-sitter based grammars for the following languages:

• CSS
• Go
• JavaScript
• EJS
• ERB
• HTML
• Python
• Ruby
• TypeScript
• CodeQL

It then falls back to using an older implementation called PrettyLights which is based heavily on an old version of TextMate's syntax highlighter. PrettyLights is what uses the grammars Linguist distributes.

You'll note that Python is in the list of tree-sitter grammars and within the list of grammars shipped with Linguist. In this case, the Python tree-sitter grammar is only used for Python. The MagicPython grammar is used for all the other languages that use this grammar but aren't Python. That means in this case you need to raise your grammar issue in the tree-sitter grammar repo. I plan to update the docs with this info when I get the chance.

It's also worth pointing out that GitHub often lags behind the upstream tree-sitter grammars as the service doesn't pull directly from the sources each and everytime the syntax highlighter is deployed so it may take some time for the upstream changes to appear on GitHub.

[verhovsky]

Thanks for the detailed answer! Just a minor correction, tree-sitter-python is the repo for the Python tree-sitter grammar. py-tree-sitter is the Python bindings for the tree sitter api.

[lildude]

🤦 whoops. Yes. That will teach me for responding from my iPad and not checking properly 😬

[mkatychev]

Hi @lildude is there any plans on extending the scope of tree-sitter language support? tree-sitter-just for instance has already handled comments that include single quotes but that's not reflected in the grammar itself: https://github.com/IndianBoy42/tree-sitter-just/blob/main/justfile#L12 it feels somewhat odd to backport these changes.

Would a commit similar to https://github.com/github-linguist/linguist/pull/6460/files be accepted to move more grammars to tree-sitter?

P.S.
It may be better to create a new discussion rather than to resurrect this one

[lildude]

Nope. This is outside of Linguist's control. The highlighting engine controls which tree-sitter grammars it uses; I just update the README.md to reflect what has been implemented by the team that maintains the highlighting engine.

There are no immediate plans to open things up for community contributions of tree-sitter grammars as it's proving to be the case that not all grammars are created equal and some have severe performance and other issues that need to be addressed upstream and many of these don't come to light until things are manually tested - many grammars don't even have tests 🫣. We don't have the resources to manually test each and every community contribution so for now it's only TextMate grammars via Linguist.

[mkatychev]

That's for the explanation! It's true that not all grammars are written equally.