tokenization.ts ×8

Frontier kind: Code frontier

unlabeled · c_cf1de19e83b2

7 tests · 4102 LOC · 21 files · introduces 0 tests · 40 LOC · 1 file

Introduces — evidence that enters the hierarchy at this concept

Code: 8 ranges40 lines · 1 files
Tests: 0 tests

Contains — complete concept membership

All code (extent): 605 ranges4102 lines · 21 files · Browse complete extent
All tests (intent): 7 testsBrowse complete intent

Neighbourhood graph

The orange circle is the focus. Violet and green circles are every ancestor and descendant, broader and narrower, at any distance; blue squares and pink diamonds are the introduced files and exact introduced tests of every visible concept, not only the focus's. Arrows point from broader to narrower concepts and bridge only concepts omitted from this view. Undirected links show source or test introduction. Concept and file size follows LOC; exact test nodes use test-count units.

Focus concept
Ancestors (broader)
Descendants (narrower)
Introduced files
Introduced tests

Introduced files, introduced tests, and structurally relevant concept specialization

In the embedded map, ordinary wheel input scrolls the page; use the visible controls to zoom and drag to pan. Open the full-screen map for canvas navigation: wheel pans, Ctrl/Command plus wheel zooms, and arrow keys pan when this region is focused. On touch screens, open the full-screen map to pan or pinch. If JavaScript or WebGL is unavailable, use the native relationship evidence on this page.

Native relationship evidence

Every exact file and test below is linked only from the concept that introduces it.

tokenization.ts ×8 — Focused concept Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×3 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme matching can match|occurrence=1
tokenization.test|title=Token theme resolving rules are inherited 2|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving rules are inherited 2|occurrence=1
tokenization.test|title=Token theme resolving same rules get merged|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving same rules get merged|occurrence=1
tokenization.test|title=Token theme resolving rules are inherited 1|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving rules are inherited 1|occurrence=1
tokenization.ts ×2 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving defaults are inherited|occurrence=1
tokenization.ts ×4 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme matching gives higher priority to deeper matches|occurrence=1
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving custom colors are first in color map|occurrence=1
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×14 — Descendant (narrower) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×28 — Descendant (narrower) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
color.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
color.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
color.ts ×2 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
color.ts ×4 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
color.ts ×55 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
encodedTokenAttributes.ts ×10 — Descendant (narrower) Introduced files:
- src/vs/editor/common/encodedTokenAttributes.ts
uri.test|title=URI URI#file, win-speciale|occurrence=1, charCode.test|title=CharCode has good values|occurrence=1, +2 — Descendant (narrower) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/charCode.test|title=CharCode has good values|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/path.test|title=Paths (Node Implementation) path|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/uri.test|title=URI File paths containing apostrophes break URI parsing and cannot be opened #276075|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/uri.test|title=URI URI#file, win-speciale|occurrence=1
utils.ts ×3 — Descendant (narrower) Introduced files:
- src/vs/base/test/common/utils.ts
lifecycle.ts ×2 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×4 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×6 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
map.ts ×97 — Descendant (narrower) Introduced files:
- src/vs/base/common/arrays.ts
- src/vs/base/common/arraysFind.ts
- src/vs/base/common/assert.ts
- src/vs/base/common/charCode.ts
- src/vs/base/common/collections.ts
- src/vs/base/common/errors.ts
- src/vs/base/common/functional.ts
- src/vs/base/common/iterator.ts
- src/vs/base/common/lifecycle.ts
- src/vs/base/common/map.ts
- src/vs/base/common/marshallingIds.ts
- src/vs/base/common/path.ts
- src/vs/base/common/platform.ts
- src/vs/base/common/process.ts
- src/vs/base/common/types.ts
- src/vs/base/common/uri.ts
- src/vs/base/test/common/utils.ts
- src/vs/nls.ts

Introduced tests

Every collected test enters the hierarchy at exactly one concept.

No tests are introduced at this concept. Its intent tests are introduced by other concepts.

Introduced code

Every collected source range enters the hierarchy at exactly one concept.

1 file ranked by introduced lines: 40 introduced LOC across 8 ranges. Expand a file to inspect source; the > gutter marks introduced lines.

src/vs/editor/common/languages/supports/tokenization.ts 40 introduced LOC · 8 ranges

Open complete file

  145
  
  	const root = new ThemeTrieElement(defaults);
  
  146
  
  	for (let i = 0, len = parsedThemeRules.length; i < len; i++) {
  
  147
  >
  		const rule = parsedThemeRules[i];
  
tokenization.ts

  148
  >
  		root.insert(rule.token, rule.fontStyle, colorMap.getId(rule.foreground), colorMap.getId(rule.background));
  
  149
  >
  	}
  
  150
  
  151
  
  	return new TokenTheme(colorMap, root);

  168
  
  	public getId(color: string | null): ColorId {
  
  169
  
  		if (color === null) {
  
  170
  >
  			return 0;
  
tokenization.ts

  171
  >
  		}
  
  172
  
  		const match = color.match(colorRegExp);
  
  173
  
  		if (!match) {

  295
  
  296
  
  	public clone(): ThemeTrieElementRule {
  
  297
  >
  		return new ThemeTrieElementRule(this._fontStyle, this._foreground, this._background);
  
tokenization.ts

  298
  >
  	}
  
  299
  
  300
  
  	public acceptOverwrite(fontStyle: FontStyle, foreground: ColorId, background: ColorId): void {
  
  301
  >
  		if (fontStyle !== FontStyle.NotSet) {
  
tokenization.ts

  302
  
  			this._fontStyle = fontStyle;
  
  303
  
  		}
  
  304
  >
  		if (foreground !== ColorId.None) {
  
tokenization.ts

  305
  >
  			this._foreground = foreground;
  
  306
  >
  		}
  
  307
  >
  		if (background !== ColorId.None) {
  
  308
  
  			this._background = background;
  
  309
  
  		}
  
  310
  >
  		this.metadata = (
  
tokenization.ts

  311
  >
  			(this._fontStyle << MetadataConsts.FONT_STYLE_OFFSET)
  
  312
  >
  			| (this._foreground << MetadataConsts.FOREGROUND_OFFSET)
  
  313
  >
  			| (this._background << MetadataConsts.BACKGROUND_OFFSET)
  
  314
  >
  		) >>> 0;
  
  315
  >
  	}
  
  316
  
  }
  
  317

  384
  
  385
  
  	public insert(token: string, fontStyle: FontStyle, foreground: ColorId, background: ColorId): void {
  
  386
  >
  		if (token === '') {
  
tokenization.ts

  387
  >
  			// Merge into the main rule
  
  388
  >
  			this._mainRule.acceptOverwrite(fontStyle, foreground, background);
  
  389
  >
  			return;
  
  390
  >
  		}
  
  391
  >
  
  392
  >
  		const dotIndex = token.indexOf('.');
  
  393
  >
  		let head: string;
  
  394
  >
  		let tail: string;
  
  395
  >
  		if (dotIndex === -1) {
  
  396
  >
  			head = token;
  
  397
  >
  			tail = '';
  
  398
  >
  		} else {
  
  399
  
  			head = token.substring(0, dotIndex);
  
  400
  
  			tail = token.substring(dotIndex + 1);
  
  401
  
  		}
  
  402
  >
  
tokenization.ts

  403
  >
  		let child = this._children.get(head);
  
  404
  >
  		if (typeof child === 'undefined') {
  
  405
  >
  			child = new ThemeTrieElement(this._mainRule.clone());
  
  406
  >
  			this._children.set(head, child);
  
  407
  >
  		}
  
  408
  >
  
  409
  >
  		child.insert(tail, fontStyle, foreground, background);
  
  410
  >
  	}
  
  411
  
  }
  
  412

tokenization.ts ×8

Introduces — evidence that enters the hierarchy at this concept

Contains — complete concept membership

Neighbourhood graph

Introduced tests

Introduced code

Direct hierarchy

Parents

Children