tokenization.ts ×14

Frontier kind: Code frontier

unlabeled · c_dce7b04d56f2

14 tests · 4050 LOC · 21 files · introduces 0 tests · 59 LOC · 1 file

Introduces — evidence that enters the hierarchy at this concept

Code: 14 ranges59 lines · 1 files
Tests: 0 tests

Contains — complete concept membership

All code (extent): 596 ranges4050 lines · 21 files · Browse complete extent
All tests (intent): 14 testsBrowse complete intent

Neighbourhood graph

The orange circle is the focus. Violet and green circles are every ancestor and descendant, broader and narrower, at any distance; blue squares and pink diamonds are the introduced files and exact introduced tests of every visible concept, not only the focus's. Arrows point from broader to narrower concepts and bridge only concepts omitted from this view. Undirected links show source or test introduction. Concept and file size follows LOC; exact test nodes use test-count units.

Focus concept
Ancestors (broader)
Descendants (narrower)
Introduced files
Introduced tests

Introduced files, introduced tests, and structurally relevant concept specialization

In the embedded map, ordinary wheel input scrolls the page; use the visible controls to zoom and drag to pan. Open the full-screen map for canvas navigation: wheel pans, Ctrl/Command plus wheel zooms, and arrow keys pan when this region is focused. On touch screens, open the full-screen map to pan or pinch. If JavaScript or WebGL is unavailable, use the native relationship evidence on this page.

Native relationship evidence

Every exact file and test below is linked only from the concept that introduces it.

tokenization.ts ×14 — Focused concept Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×3 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme matching can match|occurrence=1
tokenization.test|title=Token theme resolving rules are inherited 2|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving rules are inherited 2|occurrence=1
tokenization.test|title=Token theme resolving same rules get merged|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving same rules get merged|occurrence=1
tokenization.test|title=Token theme resolving rules are inherited 1|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving rules are inherited 1|occurrence=1
tokenization.ts ×2 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving defaults are inherited|occurrence=1
tokenization.ts ×4 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme matching gives higher priority to deeper matches|occurrence=1
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.test|title=Token theme resolving can merge incoming defaults|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving can merge incoming defaults|occurrence=1
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.test|title=Token theme resolving respects incoming defaults 4|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving respects incoming defaults 4|occurrence=1
tokenization.test|title=Token theme resolving respects incoming defaults 5|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving respects incoming defaults 5|occurrence=1
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving custom colors are first in color map|occurrence=1
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving respects incoming defaults 2|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving respects incoming defaults 3|occurrence=1
tokenization.test|title=Token theme resolving respects incoming defaults 1|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving respects incoming defaults 1|occurrence=1
tokenization.ts ×2 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×8 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×5 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/editor/test/common/modes/supports/tokenization.test|title=Token theme resolving always has defaults|occurrence=1
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×4 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×2 — Ancestor (broader) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
tokenization.ts ×28 — Descendant (narrower) Introduced files:
- src/vs/editor/common/languages/supports/tokenization.ts
color.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
color.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
color.ts ×2 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
color.ts ×4 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
color.ts ×55 — Descendant (narrower) Introduced files:
- src/vs/base/common/color.ts
encodedTokenAttributes.ts ×10 — Descendant (narrower) Introduced files:
- src/vs/editor/common/encodedTokenAttributes.ts
uri.test|title=URI URI#file, win-speciale|occurrence=1, charCode.test|title=CharCode has good values|occurrence=1, +2 — Descendant (narrower) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/charCode.test|title=CharCode has good values|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/path.test|title=Paths (Node Implementation) path|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/uri.test|title=URI File paths containing apostrophes break URI parsing and cannot be opened #276075|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/uri.test|title=URI URI#file, win-speciale|occurrence=1
utils.ts ×3 — Descendant (narrower) Introduced files:
- src/vs/base/test/common/utils.ts
lifecycle.ts ×2 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×4 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×6 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
map.ts ×97 — Descendant (narrower) Introduced files:
- src/vs/base/common/arrays.ts
- src/vs/base/common/arraysFind.ts
- src/vs/base/common/assert.ts
- src/vs/base/common/charCode.ts
- src/vs/base/common/collections.ts
- src/vs/base/common/errors.ts
- src/vs/base/common/functional.ts
- src/vs/base/common/iterator.ts
- src/vs/base/common/lifecycle.ts
- src/vs/base/common/map.ts
- src/vs/base/common/marshallingIds.ts
- src/vs/base/common/path.ts
- src/vs/base/common/platform.ts
- src/vs/base/common/process.ts
- src/vs/base/common/types.ts
- src/vs/base/common/uri.ts
- src/vs/base/test/common/utils.ts
- src/vs/nls.ts

Introduced tests

Every collected test enters the hierarchy at exactly one concept.

No tests are introduced at this concept. Its intent tests are introduced by other concepts.

Introduced code

Every collected source range enters the hierarchy at exactly one concept.

1 file ranked by introduced lines: 59 introduced LOC across 14 ranges. Expand a file to inspect source; the > gutter marks introduced lines.

src/vs/editor/common/languages/supports/tokenization.ts 59 introduced LOC · 14 ranges

Open complete file

  104
  
   * Resolve rules (i.e. inheritance).
  
  105
  
   */
  
  106
  >
  function resolveParsedTokenThemeRules(parsedThemeRules: ParsedTokenThemeRule[], customTokenColors: string[]): TokenTheme {
  
tokenization.ts

  107
  >
  
  108
  >
  	// Sort rules lexicographically, and then by index if necessary
  
  109
  >
  	parsedThemeRules.sort((a, b) => {
  
  110
  
  		const r = strcmp(a.token, b.token);
  
  111
  
  		if (r !== 0) {

  113
  
  		}
  
  114
  
  		return a.index - b.index;
  
  115
  >
  	});
  
tokenization.ts

  116
  >
  
  117
  >
  	// Determine defaults
  
  118
  >
  	let defaultFontStyle = FontStyle.None;
  
  119
  >
  	let defaultForeground = '000000';
  
  120
  >
  	let defaultBackground = 'ffffff';
  
  121
  >
  	while (parsedThemeRules.length >= 1 && parsedThemeRules[0].token === '') {
  
  122
  
  		const incomingDefaults = parsedThemeRules.shift()!;
  
  123
  
  		if (incomingDefaults.fontStyle !== FontStyle.NotSet) {

  131
  
  		}
  
  132
  
  	}
  
  133
  >
  	const colorMap = new ColorMap();
  
tokenization.ts

  134
  >
  
  135
  >
  	// start with token colors from custom token themes
  
  136
  >
  	for (const color of customTokenColors) {
  
  137
  
  		colorMap.getId(color);
  
  138
  
  	}
  
  139
  >
  
tokenization.ts

  140
  >
  
  141
  >
  	const foregroundColorId = colorMap.getId(defaultForeground);
  
  142
  >
  	const backgroundColorId = colorMap.getId(defaultBackground);
  
  143
  >
  
  144
  >
  	const defaults = new ThemeTrieElementRule(defaultFontStyle, foregroundColorId, backgroundColorId);
  
  145
  >
  	const root = new ThemeTrieElement(defaults);
  
  146
  >
  	for (let i = 0, len = parsedThemeRules.length; i < len; i++) {
  
  147
  
  		const rule = parsedThemeRules[i];
  
  148
  
  		root.insert(rule.token, rule.fontStyle, colorMap.getId(rule.foreground), colorMap.getId(rule.background));
  
  149
  
  	}
  
  150
  >
  
tokenization.ts

  151
  >
  	return new TokenTheme(colorMap, root);
  
  152
  >
  }
  
  153
  
  154
  
  const colorRegExp = /^#?([0-9A-Fa-f]{6})([0-9A-Fa-f]{2})?$/;

  161
  
  162
  
  	constructor() {
  
  163
  >
  		this._lastColorId = 0;
  
tokenization.ts

  164
  >
  		this._id2color = [];
  
  165
  >
  		this._color2id = new Map<string, ColorId>();
  
  166
  >
  	}
  
  167
  
  168
  
  	public getId(color: string | null): ColorId {
  
  169
  >
  		if (color === null) {
  
tokenization.ts

  170
  
  			return 0;
  
  171
  
  		}
  
  172
  >
  		const match = color.match(colorRegExp);
  
tokenization.ts

  173
  >
  		if (!match) {
  
  174
  
  			throw new Error('Illegal value for token color: ' + color);
  
  175
  
  		}
  
  176
  >
  		color = match[1].toUpperCase();
  
tokenization.ts

  177
  >
  		let value = this._color2id.get(color);
  
  178
  >
  		if (value) {
  
  179
  
  			return value;
  
  180
  
  		}
  
  181
  >
  		value = ++this._lastColorId;
  
tokenization.ts

  182
  >
  		this._color2id.set(color, value);
  
  183
  >
  		this._id2color[value] = Color.fromHex('#' + color);
  
  184
  >
  		return value;
  
  185
  >
  	}
  
  186
  
  187
  
  	public getColorMap(): Color[] {

  198
  
  199
  
  	public static createFromParsedTokenTheme(source: ParsedTokenThemeRule[], customTokenColors: string[]): TokenTheme {
  
  200
  >
  		return resolveParsedTokenThemeRules(source, customTokenColors);
  
tokenization.ts

  201
  >
  	}
  
  202
  
  203
  
  	private readonly _colorMap: ColorMap;

  206
  
  207
  
  	constructor(colorMap: ColorMap, root: ThemeTrieElement) {
  
  208
  >
  		this._colorMap = colorMap;
  
tokenization.ts

  209
  >
  		this._root = root;
  
  210
  >
  		this._cache = new Map<string, number>();
  
  211
  >
  	}
  
  212
  
  213
  
  	public getColorMap(): Color[] {

  284
  
  285
  
  	constructor(fontStyle: FontStyle, foreground: ColorId, background: ColorId) {
  
  286
  >
  		this._fontStyle = fontStyle;
  
tokenization.ts

  287
  >
  		this._foreground = foreground;
  
  288
  >
  		this._background = background;
  
  289
  >
  		this.metadata = (
  
  290
  >
  			(this._fontStyle << MetadataConsts.FONT_STYLE_OFFSET)
  
  291
  >
  			| (this._foreground << MetadataConsts.FOREGROUND_OFFSET)
  
  292
  >
  			| (this._background << MetadataConsts.BACKGROUND_OFFSET)
  
  293
  >
  		) >>> 0;
  
  294
  >
  	}
  
  295
  
  296
  
  	public clone(): ThemeTrieElementRule {

  344
  
  345
  
  	constructor(mainRule: ThemeTrieElementRule) {
  
  346
  >
  		this._mainRule = mainRule;
  
tokenization.ts

  347
  >
  		this._children = new Map<string, ThemeTrieElement>();
  
  348
  >
  	}
  
  349
  
  350
  
  	/**

tokenization.ts ×14

Introduces — evidence that enters the hierarchy at this concept

Contains — complete concept membership

Neighbourhood graph

Introduced tests

Introduced code

Direct hierarchy

Parents

Children