Base the CUDA lexer on C++ instead of C#3144
Open
mokashang wants to merge 1 commit into
Open
Conversation
CUDA source is compiled by nvcc, which is a C++ compiler, and modern CUDA code routinely uses C++ constructs such as templates, classes, namespaces, constexpr and nullptr. The lexer derived from CLexer, so these were highlighted as plain names rather than keywords. Derive CudaLexer from CppLexer instead. The CUDA-specific post-processing keys off Name tokens, and C++ keywords are emitted as Keyword by CppLexer, so the CUDA qualifiers, vector types, builtins and functions (e.g. __global__, float4, threadIdx, __syncthreads) are still recognized unchanged. Extend the example file with modern CUDA C++ to cover the new behavior. Fixes pygments#3127
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3127.
Problem
CUDA code is compiled by
nvcc, which is a C++ compiler, and modern CUDA routinely uses C++ constructs (templates, classes, namespaces,constexpr,nullptr,auto, etc.). However,CudaLexerderived fromCLexer, so these were highlighted as plainNametokens instead of keywords. For example,template,classandnamespaceall came out asToken.Name.Change
Derive
CudaLexerfromCppLexerinstead ofCLexer(and delegateget_tokens_unprocessedtoCppLexeraccordingly).The CUDA-specific post-processing in
get_tokens_unprocessedonly rewrites tokens that come through asName. C++ keywords are emitted asKeywordbyCppLexer, so they don't collide with the CUDA detection, and the CUDA qualifiers, vector types, builtins and functions (__global__,__restrict__,float4,dim3,threadIdx,__syncthreads, …) are still recognized exactly as before.Testing
tests/examplefiles/cuda/test.cuwith a modern CUDA C++ section (a templated__global__kernel, aclass, anamespace,constexpr/auto/nullptr) and regenerated the golden output with--update-goldens. The snapshot diff is purely additive — the existing CUDA tokens are unchanged, and the new C++ constructs are now highlighted as keywords while__global__etc. remainKeyword.Reserved.tox/pytestfull suite passes locally (the only failure is the LaTeX formatter test, which requires a workinglatexbinary and is unrelated to this change).ruff check --ignore UP031 .passes.