Vihren Atlas

tokenStore.ts ×32

Code frontier

unlabeled · c_5212bd295eb6

886 tests · 3508 LOC · 19 files · introduces 0 tests · 130 LOC · 1 file

Introduces — evidence that enters the hierarchy at this concept

Code
32 ranges130 lines · 1 files
Tests
0 tests

Contains — complete concept membership

All code (extent)
512 ranges3508 lines · 19 files · Browse complete extent
All tests (intent)
886 testsBrowse complete intent

Neighbourhood graph

The orange circle is the focus. Violet and green circles are every ancestor and descendant, broader and narrower, at any distance; blue squares and pink diamonds are the introduced files and exact introduced tests of every visible concept, not only the focus's. Arrows point from broader to narrower concepts and bridge only concepts omitted from this view. Undirected links show source or test introduction. Concept and file size follows LOC; exact test nodes use test-count units.

Introduced files, introduced tests, and structurally relevant concept specialization

In the embedded map, ordinary wheel input scrolls the page; use the visible controls to zoom and drag to pan. Open the full-screen map for canvas navigation: wheel pans, Ctrl/Command plus wheel zooms, and arrow keys pan when this region is focused. On touch screens, open the full-screen map to pan or pinch. If JavaScript or WebGL is unavailable, use the native relationship evidence on this page.

Graph controls are ready.

Interactive rendering requires JavaScript and WebGL. Use the native relationship evidence on this page while the interactive map is unavailable.

Native relationship evidence

Every exact file and test below is linked only from the concept that introduces it.

Introduced tests

Every collected test enters the hierarchy at exactly one concept.

No tests are introduced at this concept. Its intent tests are introduced by other concepts.

Introduced code

Every collected source range enters the hierarchy at exactly one concept.

1 file ranked by introduced lines: 130 introduced LOC across 32 ranges. Expand a file to inspect source; the > gutter marks introduced lines.

src/vs/editor/common/model/tokens/treeSitter/tokenStore.ts 130 introduced LOC · 32 ranges

Open complete file

1 > /*--------------------------------------------------------------------------------------------- tokenStore.ts
2 > * Copyright (c) Microsoft Corporation. All rights reserved.
3 > * Licensed under the MIT License. See License.txt in the project root for license information.
4 > *--------------------------------------------------------------------------------------------*/
5 >
6 > import { IDisposable } from '../../../../../base/common/lifecycle.js';
7 > import { ITextModel } from '../../../model.js';
8 >
9 > // Exported for tests
10 > export class ListNode implements IDisposable {
11 > parent?: ListNode;
12 > private readonly _children: Node[] = [];
13 > get children(): ReadonlyArray<Node> { return this._children; }
14 >
15 > private _length: number = 0;
16 > get length(): number { return this._length; }
17 >
18 > constructor(public readonly height: number) { }
19 >
20 > static create(node1: Node, node2: Node) {
21 const list = new ListNode(node1.height + 1);
22 list.appendChild(node1);
24 return list;
25 }
26 > tokenStore.ts
27 > canAppendChild(): boolean {
28 return this._children.length < 3;
29 }
30 > tokenStore.ts
31 > appendChild(node: Node) {
32 if (!this.canAppendChild()) {
33 throw new Error('Cannot insert more than 3 children in a ListNode');
41 }
42 }
43 > tokenStore.ts
44 > private _updateParentLength(delta: number) {
45 let updateParent = this.parent;
46 while (updateParent) {
49 }
50 }
51 > tokenStore.ts
52 > unappendChild(): Node {
53 const child = this._children.pop()!;
54 this._length -= child.length;
56 return child;
57 }
58 > tokenStore.ts
59 > prependChild(node: Node) {
60 if (this._children.length >= 3) {
61 throw new Error('Cannot prepend more than 3 children in a ListNode');
69 }
70 }
71 > tokenStore.ts
72 > unprependChild(): Node {
73 const child = this._children.shift()!;
74 this._length -= child.length;
76 return child;
77 }
78 > tokenStore.ts
79 > lastChild(): Node {
80 return this._children[this._children.length - 1];
81 }
82 > tokenStore.ts
83 > dispose() {
84 this._children.splice(0, this._children.length);
85 }
86 > } tokenStore.ts
87 >
88 > export enum TokenQuality {
89 > None = 0,
90 > ViewportGuess = 1,
91 > EditGuess = 2,
92 > Accurate = 3
93 > }
94 >
95 > type Node = ListNode | LeafNode;
96 >
97 > // Exported for tests
98 > export interface LeafNode {
99 > readonly length: number;
100 > token: number;
101 > tokenQuality: TokenQuality;
102 > height: 0;
103 > }
104 >
105 > export interface TokenUpdate {
106 > readonly startOffsetInclusive: number;
107 > readonly length: number;
108 > readonly token: number;
109 > }
110 >
111 function isLeaf(node: Node): node is LeafNode {
112 return (node as LeafNode).token !== undefined;
113 }
114 > tokenStore.ts
115 > // Heavily inspired by https://github.com/microsoft/vscode/blob/4eb2658d592cb6114a7a393655574176cc790c5b/src/vs/editor/common/model/bracketPairsTextModelPart/bracketPairsTree/concat23Trees.ts#L108-L109
116 function append(node: Node, nodeToAppend: Node): Node {
117 let curNode = node;
154 }
155 }
156 > tokenStore.ts
157 function prepend(list: Node, nodeToAppend: Node): Node {
158 let curNode = list;
188 }
189 }
190 > tokenStore.ts
191 function concat(node1: Node, node2: Node): Node {
192 if (node1.height === node2.height) {
200 }
201 }
202 > tokenStore.ts
203 > export class TokenStore implements IDisposable {
204 > private _root: Node;
205 > get root(): Node {
206 > return this._root;
207 > }
208 >
209 > constructor(private readonly _textModel: ITextModel) {
210 this._root = this.createEmptyRoot();
211 }
212 > tokenStore.ts
213 > private createEmptyRoot(): Node {
214 return {
215 length: this._textModel.getValueLength(),
219 };
220 }
221 > tokenStore.ts
222 > /**
223 > *
224 > * @param update all the tokens for the document in sequence
225 > */
226 > buildStore(tokens: TokenUpdate[], tokenQuality: TokenQuality): void {
227 this._root = this.createFromUpdates(tokens, tokenQuality);
228 }
229 > tokenStore.ts
230 > private createFromUpdates(tokens: TokenUpdate[], tokenQuality: TokenQuality): Node {
231 if (tokens.length === 0) {
232 return this.createEmptyRoot();
243 return newRoot;
244 }
245 > tokenStore.ts
246 > /**
247 > *
248 > * @param tokens tokens are in sequence in the document.
249 > */
250 > update(length: number, tokens: TokenUpdate[], tokenQuality: TokenQuality) {
251 if (tokens.length === 0) {
252 return;
254 this.replace(length, tokens[0].startOffsetInclusive, tokens, tokenQuality);
255 }
256 > tokenStore.ts
257 > delete(length: number, startOffset: number) {
258 this.replace(length, startOffset, [], TokenQuality.EditGuess);
259 }
260 > tokenStore.ts
261 > /**
262 > *
263 > * @param tokens tokens are in sequence in the document.
264 > */
265 > private replace(length: number, updateOffsetStart: number, tokens: TokenUpdate[], tokenQuality: TokenQuality) {
266 const firstUnchangedOffsetAfterUpdate = updateOffsetStart + length;
267 // Find the last unchanged node preceding the update
326 this._root = newRoot ?? this.createEmptyRoot();
327 }
328 > tokenStore.ts
329 > /**
330 > *
331 > * @param startOffsetInclusive
332 > * @param endOffsetExclusive
333 > * @param visitor Return true from visitor to exit early
334 > * @returns
335 > */
336 > private traverseInOrderInRange(startOffsetInclusive: number, endOffsetExclusive: number, visitor: (node: Node, offset: number) => boolean): void {
337 const stack: { node: Node; offset: number }[] = [{ node: this._root, offset: 0 }];
338
360 }
361 }
362 > tokenStore.ts
363 > getTokenAt(offset: number): TokenUpdate | undefined {
364 let result: TokenUpdate | undefined;
365 this.traverseInOrderInRange(offset, this._root.length, (node, offset) => {
372 return result;
373 }
374 > tokenStore.ts
375 > getTokensInRange(startOffsetInclusive: number, endOffsetExclusive: number): TokenUpdate[] {
376 const result: { token: number; startOffsetInclusive: number; length: number }[] = [];
377 this.traverseInOrderInRange(startOffsetInclusive, endOffsetExclusive, (node, offset) => {
394 return result;
395 }
396 > tokenStore.ts
397 > markForRefresh(startOffsetInclusive: number, endOffsetExclusive: number): void {
398 this.traverseInOrderInRange(startOffsetInclusive, endOffsetExclusive, (node) => {
399 if (isLeaf(node)) {
403 });
404 }
405 > tokenStore.ts
406 > rangeHasTokens(startOffsetInclusive: number, endOffsetExclusive: number, minimumTokenQuality: TokenQuality): boolean {
407 let hasAny = true;
408 this.traverseInOrderInRange(startOffsetInclusive, endOffsetExclusive, (node) => {
414 return hasAny;
415 }
416 > tokenStore.ts
417 > rangeNeedsRefresh(startOffsetInclusive: number, endOffsetExclusive: number): boolean {
418 let needsRefresh = false;
419 this.traverseInOrderInRange(startOffsetInclusive, endOffsetExclusive, (node) => {
425 return needsRefresh;
426 }
427 > tokenStore.ts
428 > getNeedsRefresh(): { startOffset: number; endOffset: number }[] {
429 const result: { startOffset: number; endOffset: number }[] = [];
430
441 return result;
442 }
443 > tokenStore.ts
444 > public deepCopy(): TokenStore {
445 const newStore = new TokenStore(this._textModel);
446 newStore._root = this._copyNodeIterative(this._root);
447 return newStore;
448 }
449 > tokenStore.ts
450 > private _copyNodeIterative(root: Node): Node {
451 const newRoot = isLeaf(root)
452 ? { length: root.length, token: root.token, tokenQuality: root.tokenQuality, height: root.height }
471 return newRoot;
472 }
473 > tokenStore.ts
474 > /**
475 > * Returns a string representation of the token tree using an iterative approach
476 > */
477 > printTree(root: Node = this._root): string {
478 const result: string[] = [];
479 const stack: Array<[Node, number]> = [[root, 0]];
496 return result.join('');
497 }
498 > tokenStore.ts
499 > dispose(): void {
500 const stack: Array<[Node, boolean]> = [[this._root, false]];
501 while (stack.length > 0) {

Direct hierarchy

Parents

Children