Vihren Atlas

textModelTokens.ts ×44

Code frontier

unlabeled · c_5a2825cdb134

863 tests · 12143 LOC · 52 files · introduces 0 tests · 336 LOC · 7 files

Introduces — evidence that enters the hierarchy at this concept

Code
84 ranges336 lines · 7 files
Tests
0 tests

Contains — complete concept membership

All code (extent)
1414 ranges12143 lines · 52 files · Browse complete extent
All tests (intent)
863 testsBrowse complete intent

Neighbourhood graph

The orange circle is the focus. Violet and green circles are every ancestor and descendant, broader and narrower, at any distance; blue squares and pink diamonds are the introduced files and exact introduced tests of every visible concept, not only the focus's. Arrows point from broader to narrower concepts and bridge only concepts omitted from this view. Undirected links show source or test introduction. Concept and file size follows LOC; exact test nodes use test-count units.

Introduced files, introduced tests, and structurally relevant concept specialization

In the embedded map, ordinary wheel input scrolls the page; use the visible controls to zoom and drag to pan. Open the full-screen map for canvas navigation: wheel pans, Ctrl/Command plus wheel zooms, and arrow keys pan when this region is focused. On touch screens, open the full-screen map to pan or pinch. If JavaScript or WebGL is unavailable, use the native relationship evidence on this page.

Graph controls are ready.

Interactive rendering requires JavaScript and WebGL. Use the native relationship evidence on this page while the interactive map is unavailable.

Native relationship evidence

Every exact file and test below is linked only from the concept that introduces it.

Introduced tests

Every collected test enters the hierarchy at exactly one concept.

No tests are introduced at this concept. Its intent tests are introduced by other concepts.

Introduced code

Every collected source range enters the hierarchy at exactly one concept.

7 files ranked by introduced lines: 336 introduced LOC across 84 ranges. Expand a file to inspect source; the > gutter marks introduced lines.

src/vs/editor/common/model/textModelTokens.ts 161 introduced LOC · 44 ranges

Open complete file

1 > /*--------------------------------------------------------------------------------------------- textModelTokens.ts
2 > * Copyright (c) Microsoft Corporation. All rights reserved.
3 > * Licensed under the MIT License. See License.txt in the project root for license information.
4 > *--------------------------------------------------------------------------------------------*/
5 >
6 > import { IdleDeadline, runWhenGlobalIdle } from '../../../base/common/async.js';
7 > import { BugIndicatingError, onUnexpectedError } from '../../../base/common/errors.js';
8 > import { setTimeout0 } from '../../../base/common/platform.js';
9 > import { StopWatch } from '../../../base/common/stopwatch.js';
10 > import { countEOL } from '../core/misc/eolCounter.js';
11 > import { LineRange } from '../core/ranges/lineRange.js';
12 > import { OffsetRange } from '../core/ranges/offsetRange.js';
13 > import { Position } from '../core/position.js';
14 > import { StandardTokenType } from '../encodedTokenAttributes.js';
15 > import { EncodedTokenizationResult, IBackgroundTokenizationStore, IBackgroundTokenizer, ILanguageIdCodec, IState, ITokenizationSupport } from '../languages.js';
16 > import { nullTokenizeEncoded } from '../languages/nullTokenize.js';
17 > import { ITextModel } from '../model.js';
18 > import { FixedArray } from './fixedArray.js';
19 > import { IModelContentChange } from './mirrorTextModel.js';
20 > import { ContiguousMultilineTokensBuilder } from '../tokens/contiguousMultilineTokensBuilder.js';
21 > import { LineTokens } from '../tokens/lineTokens.js';
22 >
23 > const enum Constants {
24 > CHEAP_TOKENIZATION_LENGTH_LIMIT = 2048
25 > }
26 >
27 > export class TokenizerWithStateStore<TState extends IState = IState> {
28 > private readonly initialState;
29 >
30 > public readonly store: TrackingTokenizationStateStore<TState>;
31 >
32 > constructor(
33 lineCount: number,
34 public readonly tokenizationSupport: ITokenizationSupport
37 this.store = new TrackingTokenizationStateStore<TState>(lineCount);
38 }
39 > textModelTokens.ts
40 > public getStartState(lineNumber: number): TState | null {
41 return this.store.getStartState(lineNumber, this.initialState);
42 }
43 > textModelTokens.ts
44 > public getFirstInvalidLine(): { lineNumber: number; startState: TState } | null {
45 return this.store.getFirstInvalidLine(this.initialState);
46 }
47 > } textModelTokens.ts
48 >
49 > export class TokenizerWithStateStoreAndTextModel<TState extends IState = IState> extends TokenizerWithStateStore<TState> {
50 > constructor(
51 lineCount: number,
52 tokenizationSupport: ITokenizationSupport,
56 super(lineCount, tokenizationSupport);
57 }
58 > textModelTokens.ts
59 > public updateTokensUntilLine(builder: ContiguousMultilineTokensBuilder, lineNumber: number): void {
60 const languageId = this._textModel.getLanguageId();
61
73 }
74 }
75 > textModelTokens.ts
76 > /** assumes state is up to date */
77 > public getTokenTypeIfInsertingCharacter(position: Position, character: string): StandardTokenType {
78 // TODO@hediet: use tokenizeLineWithEdit
79 const lineStartState = this.getStartState(position.lineNumber);
101 return lineTokens.getStandardTokenType(tokenIndex);
102 }
103 > textModelTokens.ts
104 > /** assumes state is up to date */
105 > public tokenizeLinesAt(lineNumber: number, lines: string[]): LineTokens[] | null {
106 const lineStartState: IState | null = this.getStartState(lineNumber);
107 if (!lineStartState) {
121 return result;
122 }
123 > textModelTokens.ts
124 > public hasAccurateTokensForLine(lineNumber: number): boolean {
125 const firstInvalidLineNumber = this.store.getFirstInvalidEndStateLineNumberOrMax();
126 return (lineNumber < firstInvalidLineNumber);
127 }
128 > textModelTokens.ts
129 > public isCheapToTokenize(lineNumber: number): boolean {
130 const firstInvalidLineNumber = this.store.getFirstInvalidEndStateLineNumberOrMax();
131 if (lineNumber < firstInvalidLineNumber) {
139 return false;
140 }
141 > textModelTokens.ts
142 > /**
143 > * The result is not cached.
144 > */
145 > public tokenizeHeuristically(builder: ContiguousMultilineTokensBuilder, startLineNumber: number, endLineNumber: number): { heuristicTokens: boolean } {
146 if (endLineNumber <= this.store.getFirstInvalidEndStateLineNumberOrMax()) {
147 // nothing to do
167 return { heuristicTokens: true };
168 }
169 > textModelTokens.ts
170 > private guessStartState(lineNumber: number): IState {
171 let { likelyRelevantLines, initialState } = findLikelyRelevantLines(this._textModel, lineNumber, this);
172
183 return state;
184 }
185 > } textModelTokens.ts
186 >
187 > export function findLikelyRelevantLines(model: ITextModel, lineNumber: number, store?: TokenizerWithStateStore): { likelyRelevantLines: string[]; initialState?: IState } {
188 let nonWhitespaceColumn = model.getLineFirstNonWhitespaceColumn(lineNumber);
189 const likelyRelevantLines: string[] = [];
208 return { likelyRelevantLines, initialState: initialState ?? undefined };
209 }
210 > textModelTokens.ts
211 > /**
212 > * **Invariant:**
213 > * If the text model is retokenized from line 1 to {@link getFirstInvalidEndStateLineNumber}() - 1,
214 > * then the recomputed end state for line l will be equal to {@link getEndState}(l).
215 > */
216 > export class TrackingTokenizationStateStore<TState extends IState> {
217 > private readonly _tokenizationStateStore = new TokenizationStateStore<TState>();
218 > private readonly _invalidEndStatesLineNumbers = new RangePriorityQueueImpl();
219 >
220 > constructor(private lineCount: number) {
221 this._invalidEndStatesLineNumbers.addRange(new OffsetRange(1, lineCount + 1));
222 }
223 > textModelTokens.ts
224 > public getEndState(lineNumber: number): TState | null {
225 return this._tokenizationStateStore.getEndState(lineNumber);
226 }
227 > textModelTokens.ts
228 > /**
229 > * @returns if the end state has changed.
230 > */
231 > public setEndState(lineNumber: number, state: TState): boolean {
232 if (!state) {
233 throw new BugIndicatingError('Cannot set null/undefined state');
243 return r;
244 }
245 > textModelTokens.ts
246 > public acceptChange(range: LineRange, newLineCount: number): void {
247 this.lineCount += newLineCount - range.length;
248 this._tokenizationStateStore.acceptChange(range, newLineCount);
249 this._invalidEndStatesLineNumbers.addRangeAndResize(new OffsetRange(range.startLineNumber, range.endLineNumberExclusive), newLineCount);
250 }
251 > textModelTokens.ts
252 > public acceptChanges(changes: IModelContentChange[]) {
253 for (const c of changes) {
254 const [eolCount] = countEOL(c.text);
256 }
257 }
258 > textModelTokens.ts
259 > public invalidateEndStateRange(range: LineRange): void {
260 this._invalidEndStatesLineNumbers.addRange(new OffsetRange(range.startLineNumber, range.endLineNumberExclusive));
261 }
262 > textModelTokens.ts
263 > public getFirstInvalidEndStateLineNumber(): number | null { return this._invalidEndStatesLineNumbers.min; }
264 >
265 > public getFirstInvalidEndStateLineNumberOrMax(): number {
266 return this.getFirstInvalidEndStateLineNumber() || Number.MAX_SAFE_INTEGER;
267 }
268 > textModelTokens.ts
269 > public allStatesValid(): boolean { return this._invalidEndStatesLineNumbers.min === null; }
270 >
271 > public getStartState(lineNumber: number, initialState: TState): TState | null {
272 if (lineNumber === 1) { return initialState; }
273 return this.getEndState(lineNumber - 1);
274 }
275 > textModelTokens.ts
276 > public getFirstInvalidLine(initialState: TState): { lineNumber: number; startState: TState } | null {
277 const lineNumber = this.getFirstInvalidEndStateLineNumber();
278 if (lineNumber === null) {
286 return { lineNumber, startState };
287 }
288 > } textModelTokens.ts
289 >
290 > export class TokenizationStateStore<TState extends IState> {
291 private readonly _lineEndStates = new FixedArray<TState | null>(null);
292 > textModelTokens.ts
293 > public getEndState(lineNumber: number): TState | null {
294 return this._lineEndStates.get(lineNumber);
295 }
296 > textModelTokens.ts
297 > public setEndState(lineNumber: number, state: TState): boolean {
298 const oldState = this._lineEndStates.get(lineNumber);
299 if (oldState && oldState.equals(state)) {
304 return true;
305 }
306 > textModelTokens.ts
307 > public acceptChange(range: LineRange, newLineCount: number): void {
308 let length = range.length;
309 if (newLineCount > 0 && length > 0) {
316 this._lineEndStates.replace(range.startLineNumber, length, newLineCount);
317 }
318 > textModelTokens.ts
319 > public acceptChanges(changes: IModelContentChange[]) {
320 for (const c of changes) {
321 const [eolCount] = countEOL(c.text);
323 }
324 }
325 > } textModelTokens.ts
326 >
327 > interface RangePriorityQueue {
328 > get min(): number | null;
329 > removeMin(): number | null;
330 >
331 > addRange(range: OffsetRange): void;
332 >
333 > addRangeAndResize(range: OffsetRange, newLength: number): void;
334 > }
335 >
336 > export class RangePriorityQueueImpl implements RangePriorityQueue {
337 private readonly _ranges: OffsetRange[] = [];
338 > textModelTokens.ts
339 > public getRanges(): OffsetRange[] {
340 return this._ranges;
341 }
342 > textModelTokens.ts
343 > public get min(): number | null {
344 if (this._ranges.length === 0) {
345 return null;
347 return this._ranges[0].start;
348 }
349 > textModelTokens.ts
350 > public removeMin(): number | null {
351 if (this._ranges.length === 0) {
352 return null;
360 return range.start;
361 }
362 > textModelTokens.ts
363 > public delete(value: number): void {
364 const idx = this._ranges.findIndex(r => r.contains(value));
365 if (idx !== -1) {
380 }
381 }
382 > textModelTokens.ts
383 > public addRange(range: OffsetRange): void {
384 OffsetRange.addRange(range, this._ranges);
385 }
386 > textModelTokens.ts
387 > public addRangeAndResize(range: OffsetRange, newLength: number): void {
388 let idxFirstMightBeIntersecting = 0;
389 while (!(idxFirstMightBeIntersecting >= this._ranges.length || range.start <= this._ranges[idxFirstMightBeIntersecting].endExclusive)) {
417 }
418 }
419 > textModelTokens.ts
420 > toString() {
421 return this._ranges.map(r => r.toString()).join(' + ');
422 }
423 > } textModelTokens.ts
424 >
425 >
426 function safeTokenize(languageIdCodec: ILanguageIdCodec, languageId: string, tokenizationSupport: ITokenizationSupport | null, text: string, hasEOL: boolean, state: IState): EncodedTokenizationResult {
427 let r: EncodedTokenizationResult | null = null;
442 return r;
443 }
444 > textModelTokens.ts
445 > export class DefaultBackgroundTokenizer implements IBackgroundTokenizer {
446 > private _isDisposed = false;
447 >
448 > constructor(
449 private readonly _tokenizerWithStateStore: TokenizerWithStateStoreAndTextModel,
450 private readonly _backgroundTokenStore: IBackgroundTokenizationStore,
461
462 private _isScheduled = false;
463 > private _beginBackgroundTokenization(): void { textModelTokens.ts
464 if (this._isScheduled || !this._tokenizerWithStateStore._textModel.isAttachedToEditor() || !this._hasLinesToTokenize()) {
465 return;
473 });
474 }
475 > textModelTokens.ts
476 > /**
477 > * Tokenize until the deadline occurs, but try to yield every 1-2ms.
478 > */
479 > private _backgroundTokenizeWithDeadline(deadline: IdleDeadline): void {
480 // Read the time remaining from the `deadline` immediately because it is unclear
481 // if the `deadline` object will be valid after execution leaves this function.
501 execute();
502 }
503 > textModelTokens.ts
504 > /**
505 > * Tokenize for at least 1ms.
506 > */
507 > private _backgroundTokenizeForAtLeast1ms(): void {
508 const lineCount = this._tokenizerWithStateStore._textModel.getLineCount();
509 const builder = new ContiguousMultilineTokensBuilder();
528 this.checkFinished();
529 }
530 > textModelTokens.ts
531 > private _hasLinesToTokenize(): boolean {
532 if (!this._tokenizerWithStateStore) {
533 return false;
535 return !this._tokenizerWithStateStore.store.allStatesValid();
536 }
537 > textModelTokens.ts
538 > private _tokenizeOneInvalidLine(builder: ContiguousMultilineTokensBuilder): number {
539 const firstInvalidLine = this._tokenizerWithStateStore?.getFirstInvalidLine();
540 if (!firstInvalidLine) {
544 return firstInvalidLine.lineNumber;
545 }
546 > textModelTokens.ts
547 > public checkFinished(): void {
548 if (this._isDisposed) {
549 return;
553 }
554 }
555 > textModelTokens.ts
556 > public requestTokens(startLineNumber: number, endLineNumberExclusive: number): void {
557 this._tokenizerWithStateStore.store.invalidateEndStateRange(new LineRange(startLineNumber, endLineNumberExclusive));
558 }
559 > } textModelTokens.ts
src/vs/editor/common/tokens/contiguousMultilineTokens.ts 68 introduced LOC · 14 ranges

Open complete file

1 > /*--------------------------------------------------------------------------------------------- contiguousMultilineTokens.ts
2 > * Copyright (c) Microsoft Corporation. All rights reserved.
3 > * Licensed under the MIT License. See License.txt in the project root for license information.
4 > *--------------------------------------------------------------------------------------------*/
5 >
6 > import * as arrays from '../../../base/common/arrays.js';
7 > import { readUInt32BE, writeUInt32BE } from '../../../base/common/buffer.js';
8 > import { Position } from '../core/position.js';
9 > import { IRange } from '../core/range.js';
10 > import { countEOL } from '../core/misc/eolCounter.js';
11 > import { ContiguousTokensEditing } from './contiguousTokensEditing.js';
12 > import { LineRange } from '../core/ranges/lineRange.js';
13 >
14 > /**
15 > * Represents contiguous tokens over a contiguous range of lines.
16 > */
17 > export class ContiguousMultilineTokens {
18 > public static deserialize(buff: Uint8Array, offset: number, result: ContiguousMultilineTokens[]): number {
19 const view32 = new Uint32Array(buff.buffer);
20 const startLineNumber = readUInt32BE(buff, offset); offset += 4;
29 return offset;
30 }
31 > contiguousMultilineTokens.ts
32 > /**
33 > * The start line number for this block of tokens.
34 > */
35 > private _startLineNumber: number;
36 >
37 > /**
38 > * The tokens are stored in a binary format. There is an element for each line,
39 > * so `tokens[index]` contains all tokens on line `startLineNumber + index`.
40 > *
41 > * On a specific line, each token occupies two array indices. For token i:
42 > * - at offset 2*i => endOffset
43 > * - at offset 2*i + 1 => metadata
44 > *
45 > */
46 > private _tokens: (Uint32Array | ArrayBuffer | null)[];
47 >
48 > /**
49 > * (Inclusive) start line number for these tokens.
50 > */
51 > public get startLineNumber(): number {
52 return this._startLineNumber;
53 }
54 > contiguousMultilineTokens.ts
55 > /**
56 > * (Inclusive) end line number for these tokens.
57 > */
58 > public get endLineNumber(): number {
59 return this._startLineNumber + this._tokens.length - 1;
60 }
61 > contiguousMultilineTokens.ts
62 > constructor(startLineNumber: number, tokens: Uint32Array[]) {
63 this._startLineNumber = startLineNumber;
64 this._tokens = tokens;
65 }
66 > contiguousMultilineTokens.ts
67 > getLineRange(): LineRange {
68 return new LineRange(this._startLineNumber, this._startLineNumber + this._tokens.length);
69 }
70 > contiguousMultilineTokens.ts
71 > /**
72 > * @see {@link _tokens}
73 > */
74 > public getLineTokens(lineNumber: number): Uint32Array | ArrayBuffer | null {
75 return this._tokens[lineNumber - this._startLineNumber];
76 }
77 > contiguousMultilineTokens.ts
78 > public appendLineTokens(lineTokens: Uint32Array): void {
79 this._tokens.push(lineTokens);
80 }
81 > contiguousMultilineTokens.ts
82 > public serializeSize(): number {
83 let result = 0;
84 result += 4; // 4 bytes for the start line number
94 return result;
95 }
96 > contiguousMultilineTokens.ts
97 > public serialize(destination: Uint8Array, offset: number): number {
98 writeUInt32BE(destination, this._startLineNumber, offset); offset += 4;
99 writeUInt32BE(destination, this._tokens.length, offset); offset += 4;
108 return offset;
109 }
110 > contiguousMultilineTokens.ts
111 > public applyEdit(range: IRange, text: string): void {
112 const [eolCount, firstLineLength] = countEOL(text);
113 this._acceptDeleteRange(range);
114 this._acceptInsertText(new Position(range.startLineNumber, range.startColumn), eolCount, firstLineLength);
115 }
116 > contiguousMultilineTokens.ts
117 > private _acceptDeleteRange(range: IRange): void {
118 if (range.startLineNumber === range.endLineNumber && range.startColumn === range.endColumn) {
119 // Nothing to delete
184 }
185 }
186 > contiguousMultilineTokens.ts
187 > private _acceptInsertText(position: Position, eolCount: number, firstLineLength: number): void {
188
189 if (eolCount === 0 && firstLineLength === 0) {
216 this._insertLines(position.lineNumber, eolCount);
217 }
218 > contiguousMultilineTokens.ts
219 > private _insertLines(insertIndex: number, insertCount: number): void {
220 if (insertCount === 0) {
221 return;
src/vs/editor/common/model/fixedArray.ts 27 introduced LOC · 7 ranges

Open complete file

1 > /*--------------------------------------------------------------------------------------------- fixedArray.ts
2 > * Copyright (c) Microsoft Corporation. All rights reserved.
3 > * Licensed under the MIT License. See License.txt in the project root for license information.
4 > *--------------------------------------------------------------------------------------------*/
5 >
6 > import { arrayInsert } from '../../../base/common/arrays.js';
7 >
8 > /**
9 > * An array that avoids being sparse by always
10 > * filling up unused indices with a default value.
11 > */
12 > export class FixedArray<T> {
13 > private _store: T[] = [];
14 >
15 > constructor(
16 private readonly _default: T
17 ) { }
18 > fixedArray.ts
19 > public get(index: number): T {
20 if (index < this._store.length) {
21 return this._store[index];
23 return this._default;
24 }
25 > fixedArray.ts
26 > public set(index: number, value: T): void {
27 while (index >= this._store.length) {
28 this._store[this._store.length] = this._default;
30 this._store[index] = value;
31 }
32 > fixedArray.ts
33 > public replace(index: number, oldLength: number, newLength: number): void {
34 if (index >= this._store.length) {
35 return;
49 this._store = before.concat(insertArr, after);
50 }
51 > fixedArray.ts
52 > public delete(deleteIndex: number, deleteCount: number): void {
53 if (deleteCount === 0 || deleteIndex >= this._store.length) {
54 return;
56 this._store.splice(deleteIndex, deleteCount);
57 }
58 > fixedArray.ts
59 > public insert(insertIndex: number, insertCount: number): void {
60 if (insertCount === 0 || insertIndex >= this._store.length) {
61 return;
67 this._store = arrayInsert(this._store, insertIndex, arr);
68 }
69 > } fixedArray.ts
70 >
71 function arrayFill<T>(length: number, value: T): T[] {
72 const arr: T[] = [];
src/vs/editor/common/tokens/contiguousMultilineTokensBuilder.ts 26 introduced LOC · 8 ranges

Open complete file

1 > /*--------------------------------------------------------------------------------------------- contiguousMultilineTokensBuilder.ts
2 > * Copyright (c) Microsoft Corporation. All rights reserved.
3 > * Licensed under the MIT License. See License.txt in the project root for license information.
4 > *--------------------------------------------------------------------------------------------*/
5 >
6 > import { readUInt32BE, writeUInt32BE } from '../../../base/common/buffer.js';
7 > import { ContiguousMultilineTokens } from './contiguousMultilineTokens.js';
8 >
9 > export class ContiguousMultilineTokensBuilder {
10 >
11 > public static deserialize(buff: Uint8Array): ContiguousMultilineTokens[] {
12 let offset = 0;
13 const count = readUInt32BE(buff, offset); offset += 4;
18 return result;
19 }
20 > contiguousMultilineTokensBuilder.ts
21 > private readonly _tokens: ContiguousMultilineTokens[];
22 >
23 > constructor() {
24 this._tokens = [];
25 }
26 > contiguousMultilineTokensBuilder.ts
27 > public add(lineNumber: number, lineTokens: Uint32Array): void {
28 if (this._tokens.length > 0) {
29 const last = this._tokens[this._tokens.length - 1];
36 this._tokens.push(new ContiguousMultilineTokens(lineNumber, [lineTokens]));
37 }
38 > contiguousMultilineTokensBuilder.ts
39 > public finalize(): ContiguousMultilineTokens[] {
40 return this._tokens;
41 }
42 > contiguousMultilineTokensBuilder.ts
43 > public serialize(): Uint8Array {
44 const size = this._serializeSize();
45 const result = new Uint8Array(size);
56 return result;
57 }
58 > contiguousMultilineTokensBuilder.ts
59 > private _serialize(destination: Uint8Array): void {
60 let offset = 0;
61 writeUInt32BE(destination, this._tokens.length, offset); offset += 4;
src/vs/editor/common/tokens/contiguousTokensEditing.ts 23 introduced LOC · 6 ranges

Open complete file

1 > /*--------------------------------------------------------------------------------------------- contiguousTokensEditing.ts
2 > * Copyright (c) Microsoft Corporation. All rights reserved.
3 > * Licensed under the MIT License. See License.txt in the project root for license information.
4 > *--------------------------------------------------------------------------------------------*/
5 >
6 > import { LineTokens } from './lineTokens.js';
7 >
8 > export const EMPTY_LINE_TOKENS = (new Uint32Array(0)).buffer;
9 >
10 > export class ContiguousTokensEditing {
11 >
12 > public static deleteBeginning(lineTokens: Uint32Array | ArrayBuffer | null, toChIndex: number): Uint32Array | ArrayBuffer | null {
13 if (lineTokens === null || lineTokens === EMPTY_LINE_TOKENS) {
14 return lineTokens;
16 return ContiguousTokensEditing.delete(lineTokens, 0, toChIndex);
17 }
18 > contiguousTokensEditing.ts
19 > public static deleteEnding(lineTokens: Uint32Array | ArrayBuffer | null, fromChIndex: number): Uint32Array | ArrayBuffer | null {
20 if (lineTokens === null || lineTokens === EMPTY_LINE_TOKENS) {
21 return lineTokens;
26 return ContiguousTokensEditing.delete(lineTokens, fromChIndex, lineTextLength);
27 }
28 > contiguousTokensEditing.ts
29 > public static delete(lineTokens: Uint32Array | ArrayBuffer | null, fromChIndex: number, toChIndex: number): Uint32Array | ArrayBuffer | null {
30 if (lineTokens === null || lineTokens === EMPTY_LINE_TOKENS || fromChIndex === toChIndex) {
31 return lineTokens;
83 return tmp.buffer;
84 }
85 > contiguousTokensEditing.ts
86 > public static append(lineTokens: Uint32Array | ArrayBuffer | null, _otherTokens: Uint32Array | ArrayBuffer | null): Uint32Array | ArrayBuffer | null {
87 if (_otherTokens === EMPTY_LINE_TOKENS) {
88 return lineTokens;
112 return result.buffer;
113 }
114 > contiguousTokensEditing.ts
115 > public static insert(lineTokens: Uint32Array | ArrayBuffer | null, chIndex: number, textLength: number): Uint32Array | ArrayBuffer | null {
116 if (lineTokens === null || lineTokens === EMPTY_LINE_TOKENS) {
117 // nothing to do
134 return lineTokens;
135 }
136 > } contiguousTokensEditing.ts
137 >
138 > export function toUint32Array(arr: Uint32Array | ArrayBuffer): Uint32Array<ArrayBuffer> {
139 if (arr instanceof Uint32Array) {
140 return arr as Uint32Array<ArrayBuffer>;
src/vs/editor/common/languages/nullTokenize.ts 16 introduced LOC · 4 ranges

Open complete file

1 > /*--------------------------------------------------------------------------------------------- nullTokenize.ts
2 > * Copyright (c) Microsoft Corporation. All rights reserved.
3 > * Licensed under the MIT License. See License.txt in the project root for license information.
4 > *--------------------------------------------------------------------------------------------*/
5 >
6 > import { Token, TokenizationResult, EncodedTokenizationResult, IState } from '../languages.js';
7 > import { LanguageId, FontStyle, ColorId, StandardTokenType, MetadataConsts } from '../encodedTokenAttributes.js';
8 >
9 > export const NullState: IState = new class implements IState {
10 > public clone(): IState {
11 return this;
12 }
13 > public equals(other: IState): boolean { nullTokenize.ts
14 return (this === other);
15 }
16 > }; nullTokenize.ts
17 >
18 > export function nullTokenize(languageId: string, state: IState): TokenizationResult {
19 return new TokenizationResult([new Token(0, '', languageId)], state);
20 }
21 > nullTokenize.ts
22 > export function nullTokenizeEncoded(languageId: LanguageId, state: IState | null): EncodedTokenizationResult {
23 const tokens = new Uint32Array(2);
24 tokens[0] = 0;
src/vs/editor/common/core/misc/eolCounter.ts 15 introduced LOC · 1 range

Open complete file

1 > /*--------------------------------------------------------------------------------------------- eolCounter.ts
2 > * Copyright (c) Microsoft Corporation. All rights reserved.
3 > * Licensed under the MIT License. See License.txt in the project root for license information.
4 > *--------------------------------------------------------------------------------------------*/
5 >
6 > import { CharCode } from '../../../../base/common/charCode.js';
7 >
8 > export const enum StringEOL {
9 > Unknown = 0,
10 > Invalid = 3,
11 > LF = 1,
12 > CRLF = 2
13 > }
14 >
15 > export function countEOL(text: string): [number, number, number, StringEOL] {
16 let eolCount = 0;
17 let firstLineLength = 0;

Direct hierarchy

Parents

Children