Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Commit 03aa86e

Browse files
zlaski-semmlezlaski-semmle
committed
Merge branch 'master' into cpp340a
So as to get to change-notes/1.21/analysis-cpp.md
2 parents bd13982 + 54b4e59 commit 03aa86e

285 files changed

Lines changed: 5503 additions & 1570 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎change-notes/1.20/analysis-python.md‎

Lines changed: 13 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,23 @@
11
# Improvements to Python analysis
22

3+
## General improvements
34

4-
## General improvements
5+
### Extractor changes
56

6-
> Changes that affect alerts in many files or from many queries
7-
> For example, changes to file classification
7+
The extractor now parses all Python code from a single unified grammar. This means that almost all Python code will be successfully parsed, even if mutually incompatible Python code is present in the same project. This also means that Python code for any version can be correctly parsed on a worker running any other supported version of Python. For example, Python 3.7 code is parsed correctly, even if the installed version of Python is only 3.5. This will reduce the number of syntax errors found in many projects.
8+
9+
### Regular expression analysis improvements
10+
11+
The Python `re` (regular expressions) module library has a couple of constants called `MULTILINE` and `VERBOSE` which determine the parsing of regular expressions. Python 3.6 changed the implementation of these constants, which resulted in false positive results for some queries. The relevant QL libraries have been updated to support both implementations which will remove false positive results from projects that use Python 3.6 and later versions.
12+
13+
### API improvements
814

9-
The constants `MULTILINE` and `VERBOSE` in `re` module, are now understood for Python 3.6 and upward.
10-
Removes false positives seen when using Python 3.6, but not when using earlier versions.
1115
The API has been improved to declutter the global namespace and improve discoverability and readability.
1216
* New predicates `ModuleObject::named(name)` and `ModuleObject.attr(name)` have been added, allowing more readable access to common objects. For example, `(any ModuleObject m | m.getName() = "sys").getAttribute("exit")` can be replaced with `ModuleObject::named("sys").attr("exit")`
13-
* The API for accessing builtin functions has been improved. Predicates of the form `theXXXFunction()`, such as `theLenFunction()`, have been deprecated in favour of `Object::builtin(name)`.
17+
* The API for accessing builtin functions has been improved. Predicates of the form `theXXXFunction()`, such as `theLenFunction()`, have been deprecated in favor of `Object::builtin(name)`.
1418
* A configuration based API has been added for writing data flow and taint tracking queries. This is provided as a convenience for query authors who have written data flow or taint tracking queries for other languages, so they can use a similar format of query across multiple languages.
1519

16-
## New queries
20+
## New queries
1721

1822
| **Query** | **Tags** | **Purpose** |
1923
|-----------------------------|-----------|--------------------------------------------------------------------|
@@ -24,7 +28,7 @@ The API has been improved to declutter the global namespace and improve discover
2428
| Overly permissive file permissions (`py/overly-permissive-file`) | security, external/cwe/cwe-732 | Finds instances where a file is created with overly permissive permissions. Results are not shown on LGTM by default. |
2529
| Use of insecure SSL/TLS version (`py/insecure-protocol`) | security, external/cwe/cwe-327 | Finds instances where a known insecure protocol has been specified. Results are shown on LGTM by default. |
2630

27-
## Changes to existing queries
31+
## Changes to existing queries
2832

2933
| **Query** | **Expected impact** | **Change** |
3034
|----------------------------|------------------------|------------------------------------------------------------------|
@@ -35,11 +39,8 @@ The API has been improved to declutter the global namespace and improve discover
3539
| Unused import (`py/unused-import`) | Fewer false positive results | Results where the imported module is used in a `doctest` string are no longer reported. |
3640
| Unused import (`py/unused-import`) | Fewer false positive results | Results where the imported module is used in a type-hint comment are no longer reported. |
3741

38-
## Changes to code extraction
39-
40-
* The extractor now parses all Python code from a single unified grammar. This means that almost all Python code will be successfully parsed, even if mutually incompatible Python code is present in the same project. This also means that Python code for any version can be correctly parsed on a worker running any other supported version of Python. For example, Python 3.7 code is parsed correctly, even if the installed version of Python is only 3.5.
4142

42-
## Changes to QL libraries
43+
## Changes to QL libraries
4344

4445
* Added support for the `dill` pickle library.
4546
* Added support for the `bottle` web framework.

‎change-notes/1.20/extractor-javascript.md‎

Lines changed: 6 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -2,25 +2,11 @@
22

33
# Improvements to JavaScript analysis
44

5-
> NOTES
6-
>
7-
> Please describe your changes in terms that are suitable for
8-
> customers to read. These notes will have only minor tidying up
9-
> before they are published as part of the release notes.
10-
>
11-
> This file is written for lgtm users and should contain *only*
12-
> notes about changes that affect lgtm enterprise users. Add
13-
> any other customer-facing changes to the `studio-java.md`
14-
> file.
15-
>
16-
17-
## General improvements
18-
195
## Changes to code extraction
206

21-
* Parallel extraction of JavaScript files (but not TypeScript files) on LGTM is now supported. The `LGTM_THREADS` environment variable can be set to indicate how many files should be extracted in parallel. If this variable is not set, parallel extraction is disabled.
22-
* The extractor now offers experimental support for [E4X](https://developer.mozilla.org/en-US/docs/Archive/Web/E4X), a legacy language extension developed by Mozilla.
23-
* The extractor now supports additional [Flow](https://flow.org/) syntax.
24-
* The extractor now supports [Nullish Coalescing](https://github.com/tc39/proposal-nullish-coalescing) expressions.
25-
* The extractor now supports [TypeScript 3.2](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-2.html).
26-
* The TypeScript extractor now handles the control-flow of logical operators and destructuring assignments more accurately.
7+
* Parallel extraction of JavaScript files (but not TypeScript files) on LGTM is now supported. If LGTM is configured to evaluate queries using multiple threads, then JavaScript files are also extracted using multiple threads.
8+
* Experimental support for [E4X](https://developer.mozilla.org/en-US/docs/Archive/Web/E4X), a legacy language extension developed by Mozilla, is available.
9+
* Additional [Flow](https://flow.org/) syntax is now supported.
10+
* [Nullish Coalescing](https://github.com/tc39/proposal-nullish-coalescing) expressions are now supported.
11+
* [TypeScript 3.2](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-2.html) is now supported.
12+
* The TypeScript extractor now handles the control flow of logical operators and destructuring assignments more accurately.

‎change-notes/1.20/support/versions-compilers.csv‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,4 @@ Java,"Java 6 to 11 [2]_.","javac (OpenJDK and Oracle JDK)
1313
Eclipse compiler for Java (ECJ) batch compiler",``.java``
1414
JavaScript,ECMAScript 2018 or lower,Not applicable,"``.js``, ``.jsx``, ``.mjs``, ``.es``, ``.es6``, ``.htm``, ``.html``, ``.xhm``, ``.xhtml``, ``.vue``, ``.json`` [3]_."
1515
Python,"2.7, 3.5, 3.6, 3.7",Not applicable,``.py``
16-
TypeScript [4]_.,"2.6, 2.7, 2.8, 2.9, 3.0, 3.1",Standard TypeScript compiler,"``.ts``, ``.tsx``"
16+
TypeScript [4]_.,"2.6, 2.7, 2.8, 2.9, 3.0, 3.1, 3.2",Standard TypeScript compiler,"``.ts``, ``.tsx``"

‎change-notes/1.21/analysis-cpp.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Improvements to C/C++ analysis
2+
3+
## General improvements
4+
5+
## New queries
6+
7+
| **Query** | **Tags** | **Purpose** |
8+
|-----------------------------|-----------|--------------------------------------------------------------------|
9+
10+
## Changes to existing queries
11+
12+
| **Query** | **Expected impact** | **Change** |
13+
|----------------------------|------------------------|------------------------------------------------------------------|
14+
| Mismatching new/free or malloc/delete (`cpp/new-free-mismatch`) | Fewer false positive results | Fixed an issue where functions were being identified as allocation functions inappropriately. Also affects `cpp/new-array-delete-mismatch` and `cpp/new-delete-array-mismatch`. |
15+
| Overflow in uncontrolled allocation size (`cpp/uncontrolled-allocation-size`) | More correct results | This query has been reworked so that it can find a wider variety of results. |
16+
| Memory may not be freed (`cpp/memory-may-not-be-freed`) | More correct results | Support added for more Microsoft-specific allocation functions, including `LocalAlloc`, `GlobalAlloc`, `HeapAlloc` and `CoTaskMemAlloc`. |
17+
| Memory is never freed (`cpp/memory-never-freed`) | More correct results | Support added for more Microsoft-specific allocation functions, including `LocalAlloc`, `GlobalAlloc`, `HeapAlloc` and `CoTaskMemAlloc`. |
18+
| Resource not released in destructor (`cpp/resource-not-released-in-destructor`) | Fewer false positive results | Resource allocation and deallocation functions are now determined more accurately. |
19+
| Comparison result is always the same | Fewer false positive results | The range analysis library is now more conservative about floating point values being possibly `NaN` |
20+
| Wrong type of arguments to formatting function (`cpp/wrong-type-format-argument`) | More correct results and fewer false positive results | This query now more accurately identifies wide and non-wide string/character format arguments on different platforms. Platform detection has also been made more accurate for the purposes of this query. |
21+
22+
## Changes to QL libraries

‎change-notes/1.21/analysis-csharp.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# Improvements to C# analysis
2+
3+
## Changes to existing queries
4+
5+
| **Query** | **Expected impact** | **Change** |
6+
|------------------------------|------------------------|-----------------------------------|
7+
8+
9+
## Changes to code extraction
10+
11+
* Named attribute arguments are now extracted.
12+
13+
## Changes to QL libraries
14+
15+
* The class `Attribute` has two new predicates: `getConstructorArgument()` and `getNamedArgument()`. The first predicate returns arguments to the underlying constructor call and the latter returns named arguments for initializing fields and properties.
16+
17+
## Changes to autobuilder

‎change-notes/1.21/analysis-javascript.md‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,15 @@
1717

1818
| **Query** | **Expected impact** | **Change** |
1919
|--------------------------------|------------------------------|---------------------------------------------------------------------------|
20+
| Arbitrary file write during zip extraction ("Zip Slip") | More results | This rule now considers more libraries, including tar as well as zip. |
21+
| Client-side URL redirect | More results and fewer false-positive results | This rule now recognizes additional uses of the document URL. This rule now treats URLs as safe in more cases where the hostname cannot be tampered with. |
22+
| Double escaping or unescaping | More results | This rule now considers the flow of regular expressions literals. |
2023
| Expression has no effect | Fewer false-positive results | This rule now treats uses of `Object.defineProperty` more conservatively. |
24+
| Incomplete string escaping or encoding | More results | This rule now considers the flow of regular expressions literals. |
25+
| Replacement of a substring with itself | More results | This rule now considers the flow of regular expressions literals. |
26+
| Server-side URL redirect | Fewer false-positive results | This rule now treats URLs as safe in more cases where the hostname cannot be tampered with. |
2127
| Useless assignment to property | Fewer false-positive results | This rule now ignore reads of additional getters. |
22-
| Arbitrary file write during zip extraction ("Zip Slip") | More results | This rule now considers more libraries, including tar as well as zip. |
2328

2429
## Changes to QL libraries
30+
31+
* `RegExpLiteral` is now a `DataFlow::SourceNode`.

‎cpp/config/suites/security/cwe-119‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,8 @@
33
@name Call to memory access function may overflow buffer (CWE-119)
44
+ semmlecode-cpp-queries/Critical/OverflowStatic.ql: /CWE/CWE-119
55
@name Static array access may cause overflow (CWE-119)
6-
# + semmlecode-cpp-queries/Critical/OverflowDestination.ql: /CWE/CWE-119
7-
# ^ disabled due to timeout issue
6+
+ semmlecode-cpp-queries/Critical/OverflowDestination.ql: /CWE/CWE-119
7+
@name Copy function using source size (CWE-119)
88
+ semmlecode-cpp-queries/Likely Bugs/Memory Management/SuspiciousCallToStrncat.ql: /CWE/CWE-119
99
@name Potentially unsafe call to strncat (CWE-119)
1010
+ semmlecode-cpp-queries/Likely Bugs/Memory Management/StrncpyFlippedArgs.ql: /CWE/CWE-119

‎cpp/ql/src/Critical/NewDelete.qll‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@
33
*/
44
import cpp
55
import semmle.code.cpp.controlflow.SSA
6+
import semmle.code.cpp.dataflow.DataFlow
67

78
/**
89
* Holds if `alloc` is a use of `malloc` or `new`. `kind` is
@@ -46,7 +47,10 @@ predicate allocExprOrIndirect(Expr alloc, string kind) {
4647
alloc.(FunctionCall).getTarget() = rtn.getEnclosingFunction() and
4748
(
4849
allocExprOrIndirect(rtn.getExpr(), kind) or
49-
allocReaches0(rtn.getExpr(), _, kind)
50+
exists(Expr e |
51+
allocExprOrIndirect(e, kind) and
52+
DataFlow::localFlow(DataFlow::exprNode(e), DataFlow::exprNode(rtn.getExpr()))
53+
)
5054
)
5155
)
5256
}

‎cpp/ql/src/Likely Bugs/Arithmetic/PointlessComparison.ql‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
* readability
1313
*/
1414
import cpp
15+
private import semmle.code.cpp.commons.Exclusions
1516
private import semmle.code.cpp.rangeanalysis.PointlessComparison
1617
private import semmle.code.cpp.rangeanalysis.RangeAnalysisUtils
1718
import UnsignedGEZero
@@ -31,6 +32,7 @@ from
3132
where
3233
not cmp.isInMacroExpansion() and
3334
not cmp.isFromTemplateInstantiation(_) and
35+
not functionContainsDisabledCode(cmp.getEnclosingFunction()) and
3436
reachablePointlessComparison(cmp, left, right, value, ss) and
3537

3638
// a comparison between an enum and zero is always valid because whether

‎cpp/ql/src/Likely Bugs/Likely Typos/ExprHasNoEffect.ql‎

Lines changed: 9 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
* external/cwe/cwe-561
1212
*/
1313
import cpp
14+
private import semmle.code.cpp.commons.Exclusions
1415

1516
class PureExprInVoidContext extends ExprInVoidContext {
1617
PureExprInVoidContext() { this.isPure() }
@@ -23,71 +24,29 @@ predicate accessInInitOfForStmt(Expr e) {
2324
s.getExpr() = e)
2425
}
2526

26-
/**
27-
* Holds if the preprocessor branch `pbd` is on line `pbdStartLine` in file `file`.
28-
*/
29-
predicate pbdLocation(PreprocessorBranchDirective pbd, string file, int pbdStartLine) {
30-
pbd.getLocation().hasLocationInfo(file, pbdStartLine, _, _, _)
31-
}
32-
33-
/**
34-
* Holds if the body of the function `f` is on lines `fBlockStartLine` to `fBlockEndLine` in file `file`.
35-
*/
36-
predicate functionLocation(Function f, string file, int fBlockStartLine, int fBlockEndLine) {
37-
f.getBlock().getLocation().hasLocationInfo(file, fBlockStartLine, _, fBlockEndLine, _)
38-
}
3927
/**
4028
* Holds if the function `f`, or a function called by it, contains
4129
* code excluded by the preprocessor.
4230
*/
43-
predicate containsDisabledCode(Function f) {
44-
// `f` contains a preprocessor branch that was not taken
45-
exists(PreprocessorBranchDirective pbd, string file, int pbdStartLine, int fBlockStartLine, int fBlockEndLine |
46-
functionLocation(f, file, fBlockStartLine, fBlockEndLine) and
47-
pbdLocation(pbd, file, pbdStartLine) and
48-
pbdStartLine <= fBlockEndLine and
49-
pbdStartLine >= fBlockStartLine and
50-
(
51-
pbd.(PreprocessorBranch).wasNotTaken() or
52-
53-
// an else either was not taken, or it's corresponding branch
54-
// was not taken.
55-
pbd instanceof PreprocessorElse
56-
)
57-
) or
58-
31+
predicate functionContainsDisabledCodeRecursive(Function f) {
32+
functionContainsDisabledCode(f) or
5933
// recurse into function calls
6034
exists(FunctionCall fc |
6135
fc.getEnclosingFunction() = f and
62-
containsDisabledCode(fc.getTarget())
36+
functionContainsDisabledCodeRecursive(fc.getTarget())
6337
)
6438
}
6539

66-
6740
/**
6841
* Holds if the function `f`, or a function called by it, is inside a
6942
* preprocessor branch that may have code in another arm
7043
*/
71-
predicate definedInIfDef(Function f) {
72-
exists(PreprocessorBranchDirective pbd, string file, int pbdStartLine, int pbdEndLine, int fBlockStartLine, int fBlockEndLine |
73-
functionLocation(f, file, fBlockStartLine, fBlockEndLine) and
74-
pbdLocation(pbd, file, pbdStartLine) and
75-
pbdLocation(pbd.getNext(), file, pbdEndLine) and
76-
pbdStartLine <= fBlockStartLine and
77-
pbdEndLine >= fBlockEndLine and
78-
// pbd is a preprocessor branch where multiple branches exist
79-
(
80-
pbd.getNext() instanceof PreprocessorElse or
81-
pbd instanceof PreprocessorElse or
82-
pbd.getNext() instanceof PreprocessorElif or
83-
pbd instanceof PreprocessorElif
84-
)
85-
) or
86-
44+
predicate functionDefinedInIfDefRecursive(Function f) {
45+
functionDefinedInIfDef(f) or
8746
// recurse into function calls
8847
exists(FunctionCall fc |
8948
fc.getEnclosingFunction() = f and
90-
definedInIfDef(fc.getTarget())
49+
functionDefinedInIfDefRecursive(fc.getTarget())
9150
)
9251
}
9352

@@ -121,8 +80,8 @@ where // EQExprs are covered by CompareWhereAssignMeant.ql
12180
not parent instanceof PureExprInVoidContext and
12281
not peivc.getEnclosingFunction().isCompilerGenerated() and
12382
not peivc.getType() instanceof UnknownType and
124-
not containsDisabledCode(peivc.(FunctionCall).getTarget()) and
125-
not definedInIfDef(peivc.(FunctionCall).getTarget()) and
83+
not functionContainsDisabledCodeRecursive(peivc.(FunctionCall).getTarget()) and
84+
not functionDefinedInIfDefRecursive(peivc.(FunctionCall).getTarget()) and
12685
if peivc instanceof FunctionCall then
12786
exists(Function target |
12887
target = peivc.(FunctionCall).getTarget() and

0 commit comments

Comments
 (0)