Vihren Atlas

lineTokens.ts ×55

Code frontier

unlabeled · c_a617ef8de527

885 tests · 7393 LOC · 34 files · introduces 0 tests · 214 LOC · 1 file

Introduces — evidence that enters the hierarchy at this concept

Code
55 ranges214 lines · 1 files
Tests
0 tests

Contains — complete concept membership

All code (extent)
1111 ranges7393 lines · 34 files · Browse complete extent
All tests (intent)
885 testsBrowse complete intent

Neighbourhood graph

The orange circle is the focus. Violet and green circles are every ancestor and descendant, broader and narrower, at any distance; blue squares and pink diamonds are the introduced files and exact introduced tests of every visible concept, not only the focus's. Arrows point from broader to narrower concepts and bridge only concepts omitted from this view. Undirected links show source or test introduction. Concept and file size follows LOC; exact test nodes use test-count units.

Introduced files, introduced tests, and structurally relevant concept specialization

In the embedded map, ordinary wheel input scrolls the page; use the visible controls to zoom and drag to pan. Open the full-screen map for canvas navigation: wheel pans, Ctrl/Command plus wheel zooms, and arrow keys pan when this region is focused. On touch screens, open the full-screen map to pan or pinch. If JavaScript or WebGL is unavailable, use the native relationship evidence on this page.

Graph controls are ready.

Interactive rendering requires JavaScript and WebGL. Use the native relationship evidence on this page while the interactive map is unavailable.

Native relationship evidence

Every exact file and test below is linked only from the concept that introduces it.

Introduced tests

Every collected test enters the hierarchy at exactly one concept.

No tests are introduced at this concept. Its intent tests are introduced by other concepts.

Introduced code

Every collected source range enters the hierarchy at exactly one concept.

1 file ranked by introduced lines: 214 introduced LOC across 55 ranges. Expand a file to inspect source; the > gutter marks introduced lines.

src/vs/editor/common/tokens/lineTokens.ts 214 introduced LOC · 55 ranges

Open complete file

1 > /*--------------------------------------------------------------------------------------------- lineTokens.ts
2 > * Copyright (c) Microsoft Corporation. All rights reserved.
3 > * Licensed under the MIT License. See License.txt in the project root for license information.
4 > *--------------------------------------------------------------------------------------------*/
5 >
6 > import { ILanguageIdCodec } from '../languages.js';
7 > import { FontStyle, ColorId, StandardTokenType, MetadataConsts, ITokenPresentation, TokenMetadata } from '../encodedTokenAttributes.js';
8 > import { IPosition } from '../core/position.js';
9 > import { ITextModel } from '../model.js';
10 > import { OffsetRange } from '../core/ranges/offsetRange.js';
11 > import { onUnexpectedError } from '../../../base/common/errors.js';
12 >
13 >
14 > export interface IViewLineTokens {
15 > languageIdCodec: ILanguageIdCodec;
16 > equals(other: IViewLineTokens): boolean;
17 > getCount(): number;
18 > getStandardTokenType(tokenIndex: number): StandardTokenType;
19 > getForeground(tokenIndex: number): ColorId;
20 > getEndOffset(tokenIndex: number): number;
21 > getClassName(tokenIndex: number): string;
22 > getInlineStyle(tokenIndex: number, colorMap: string[]): string;
23 > getPresentation(tokenIndex: number): ITokenPresentation;
24 > findTokenIndexAtOffset(offset: number): number;
25 > getLineContent(): string;
26 > getMetadata(tokenIndex: number): number;
27 > getLanguageId(tokenIndex: number): string;
28 > getTokenText(tokenIndex: number): string;
29 > forEach(callback: (tokenIndex: number) => void): void;
30 > }
31 >
32 > export class LineTokens implements IViewLineTokens {
33 > public static createEmpty(lineContent: string, decoder: ILanguageIdCodec): LineTokens {
34 > const defaultMetadata = LineTokens.defaultTokenMetadata;
35 >
36 > const tokens = new Uint32Array(2);
37 > tokens[0] = lineContent.length;
38 > tokens[1] = defaultMetadata;
39 >
40 > return new LineTokens(tokens, lineContent, decoder);
41 > }
42 >
43 > public static createFromTextAndMetadata(data: { text: string; metadata: number }[], decoder: ILanguageIdCodec): LineTokens {
44 let offset: number = 0;
45 let fullText: string = '';
52 return new LineTokens(new Uint32Array(tokens), fullText, decoder);
53 }
54 > lineTokens.ts
55 > public static convertToEndOffset(tokens: Uint32Array, lineTextLength: number): void {
56 const tokenCount = (tokens.length >>> 1);
57 const lastTokenIndex = tokenCount - 1;
61 tokens[lastTokenIndex << 1] = lineTextLength;
62 }
63 > lineTokens.ts
64 > public static findIndexInTokensArray(tokens: Uint32Array, desiredIndex: number): number {
65 if (tokens.length <= 2) {
66 return 0;
86 return low;
87 }
88 > lineTokens.ts
89 > _lineTokensBrand: void = undefined;
90 >
91 > private readonly _tokens: Uint32Array;
92 > private readonly _tokensCount: number;
93 > private readonly _text: string;
94 >
95 > public readonly languageIdCodec: ILanguageIdCodec;
96 >
97 > public static defaultTokenMetadata = (
98 > (FontStyle.None << MetadataConsts.FONT_STYLE_OFFSET)
99 > | (ColorId.DefaultForeground << MetadataConsts.FOREGROUND_OFFSET)
100 > | (ColorId.DefaultBackground << MetadataConsts.BACKGROUND_OFFSET)
101 > ) >>> 0;
102 >
103 > constructor(tokens: Uint32Array, text: string, decoder: ILanguageIdCodec) {
104 const tokensLength = tokens.length > 1 ? tokens[tokens.length - 2] : 0;
105 if (tokensLength !== text.length) {
111 this.languageIdCodec = decoder;
112 }
113 > lineTokens.ts
114 > public getTextLength(): number {
115 return this._text.length;
116 }
117 > lineTokens.ts
118 > public equals(other: IViewLineTokens): boolean {
119 if (other instanceof LineTokens) {
120 return this.slicedEquals(other, 0, this._tokensCount);
122 return false;
123 }
124 > lineTokens.ts
125 > public slicedEquals(other: LineTokens, sliceFromTokenIndex: number, sliceTokenCount: number): boolean {
126 if (this._text !== other._text) {
127 return false;
139 return true;
140 }
141 > lineTokens.ts
142 > public getLineContent(): string {
143 return this._text;
144 }
145 > lineTokens.ts
146 > public getCount(): number {
147 return this._tokensCount;
148 }
149 > lineTokens.ts
150 > public getStartOffset(tokenIndex: number): number {
151 if (tokenIndex > 0) {
152 return this._tokens[(tokenIndex - 1) << 1];
154 return 0;
155 }
156 > lineTokens.ts
157 > public getMetadata(tokenIndex: number): number {
158 const metadata = this._tokens[(tokenIndex << 1) + 1];
159 return metadata;
160 }
161 > lineTokens.ts
162 > public getLanguageId(tokenIndex: number): string {
163 const metadata = this._tokens[(tokenIndex << 1) + 1];
164 const languageId = TokenMetadata.getLanguageId(metadata);
165 return this.languageIdCodec.decodeLanguageId(languageId);
166 }
167 > lineTokens.ts
168 > public getStandardTokenType(tokenIndex: number): StandardTokenType {
169 const metadata = this._tokens[(tokenIndex << 1) + 1];
170 return TokenMetadata.getTokenType(metadata);
171 }
172 > lineTokens.ts
173 > public getForeground(tokenIndex: number): ColorId {
174 const metadata = this._tokens[(tokenIndex << 1) + 1];
175 return TokenMetadata.getForeground(metadata);
176 }
177 > lineTokens.ts
178 > public getClassName(tokenIndex: number): string {
179 const metadata = this._tokens[(tokenIndex << 1) + 1];
180 return TokenMetadata.getClassNameFromMetadata(metadata);
181 }
182 > lineTokens.ts
183 > public getInlineStyle(tokenIndex: number, colorMap: string[]): string {
184 const metadata = this._tokens[(tokenIndex << 1) + 1];
185 return TokenMetadata.getInlineStyleFromMetadata(metadata, colorMap);
186 }
187 > lineTokens.ts
188 > public getPresentation(tokenIndex: number): ITokenPresentation {
189 const metadata = this._tokens[(tokenIndex << 1) + 1];
190 return TokenMetadata.getPresentationFromMetadata(metadata);
191 }
192 > lineTokens.ts
193 > public getEndOffset(tokenIndex: number): number {
194 return this._tokens[tokenIndex << 1];
195 }
196 > lineTokens.ts
197 > /**
198 > * Find the token containing offset `offset`.
199 > * @param offset The search offset
200 > * @return The index of the token containing the offset.
201 > */
202 > public findTokenIndexAtOffset(offset: number): number {
203 return LineTokens.findIndexInTokensArray(this._tokens, offset);
204 }
205 > lineTokens.ts
206 > public inflate(): IViewLineTokens {
207 return this;
208 }
209 > lineTokens.ts
210 > public sliceAndInflate(startOffset: number, endOffset: number, deltaOffset: number): IViewLineTokens {
211 return new SliceLineTokens(this, startOffset, endOffset, deltaOffset);
212 }
213 > lineTokens.ts
214 > public sliceZeroCopy(range: OffsetRange): IViewLineTokens {
215 return this.sliceAndInflate(range.start, range.endExclusive, 0);
216 }
217 > lineTokens.ts
218 > /**
219 > * @pure
220 > * @param insertTokens Must be sorted by offset.
221 > */
222 > public withInserted(insertTokens: { offset: number; text: string; tokenMetadata: number }[]): LineTokens {
223 if (insertTokens.length === 0) {
224 return this;
262 return new LineTokens(new Uint32Array(newTokens), text, this.languageIdCodec);
263 }
264 > lineTokens.ts
265 > public getTokensInRange(range: OffsetRange): TokenArray {
266 const builder = new TokenArrayBuilder();
267
279 return builder.build();
280 }
281 > lineTokens.ts
282 > public getTokenText(tokenIndex: number): string {
283 const startOffset = this.getStartOffset(tokenIndex);
284 const endOffset = this.getEndOffset(tokenIndex);
286 return text;
287 }
288 > lineTokens.ts
289 > public forEach(callback: (tokenIndex: number) => void): void {
290 const tokenCount = this.getCount();
291 for (let tokenIndex = 0; tokenIndex < tokenCount; tokenIndex++) {
293 }
294 }
295 > lineTokens.ts
296 > toString(): string {
297 let result = '';
298 this.forEach((i) => {
301 return result;
302 }
303 > } lineTokens.ts
304 >
305 > class SliceLineTokens implements IViewLineTokens {
306 >
307 > private readonly _source: LineTokens;
308 > private readonly _startOffset: number;
309 > private readonly _endOffset: number;
310 > private readonly _deltaOffset: number;
311 >
312 > private readonly _firstTokenIndex: number;
313 > private readonly _tokensCount: number;
314 >
315 > public readonly languageIdCodec: ILanguageIdCodec;
316 >
317 > constructor(source: LineTokens, startOffset: number, endOffset: number, deltaOffset: number) {
318 this._source = source;
319 this._startOffset = startOffset;
332 }
333 }
334 > lineTokens.ts
335 > public getMetadata(tokenIndex: number): number {
336 return this._source.getMetadata(this._firstTokenIndex + tokenIndex);
337 }
338 > lineTokens.ts
339 > public getLanguageId(tokenIndex: number): string {
340 return this._source.getLanguageId(this._firstTokenIndex + tokenIndex);
341 }
342 > lineTokens.ts
343 > public getLineContent(): string {
344 return this._source.getLineContent().substring(this._startOffset, this._endOffset);
345 }
346 > lineTokens.ts
347 > public equals(other: IViewLineTokens): boolean {
348 if (other instanceof SliceLineTokens) {
349 return (
356 return false;
357 }
358 > lineTokens.ts
359 > public getCount(): number {
360 return this._tokensCount;
361 }
362 > lineTokens.ts
363 > public getStandardTokenType(tokenIndex: number): StandardTokenType {
364 return this._source.getStandardTokenType(this._firstTokenIndex + tokenIndex);
365 }
366 > lineTokens.ts
367 > public getForeground(tokenIndex: number): ColorId {
368 return this._source.getForeground(this._firstTokenIndex + tokenIndex);
369 }
370 > lineTokens.ts
371 > public getEndOffset(tokenIndex: number): number {
372 const tokenEndOffset = this._source.getEndOffset(this._firstTokenIndex + tokenIndex);
373 return Math.min(this._endOffset, tokenEndOffset) - this._startOffset + this._deltaOffset;
374 }
375 > lineTokens.ts
376 > public getClassName(tokenIndex: number): string {
377 return this._source.getClassName(this._firstTokenIndex + tokenIndex);
378 }
379 > lineTokens.ts
380 > public getInlineStyle(tokenIndex: number, colorMap: string[]): string {
381 return this._source.getInlineStyle(this._firstTokenIndex + tokenIndex, colorMap);
382 }
383 > lineTokens.ts
384 > public getPresentation(tokenIndex: number): ITokenPresentation {
385 return this._source.getPresentation(this._firstTokenIndex + tokenIndex);
386 }
387 > lineTokens.ts
388 > public findTokenIndexAtOffset(offset: number): number {
389 return this._source.findTokenIndexAtOffset(offset + this._startOffset - this._deltaOffset) - this._firstTokenIndex;
390 }
391 > lineTokens.ts
392 > public getTokenText(tokenIndex: number): string {
393 const adjustedTokenIndex = this._firstTokenIndex + tokenIndex;
394 const tokenStartOffset = this._source.getStartOffset(adjustedTokenIndex);
403 return text;
404 }
405 > lineTokens.ts
406 > public forEach(callback: (tokenIndex: number) => void): void {
407 for (let tokenIndex = 0; tokenIndex < this.getCount(); tokenIndex++) {
408 callback(tokenIndex);
409 }
410 }
411 > } lineTokens.ts
412 >
413 > export function getStandardTokenTypeAtPosition(model: ITextModel, position: IPosition): StandardTokenType | undefined {
414 const lineNumber = position.lineNumber;
415 if (!model.tokenization.isCheapToTokenize(lineNumber)) {
422 return tokenType;
423 }
424 > lineTokens.ts
425 >
426 >
427 > /**
428 > * This class represents a sequence of tokens.
429 > * Conceptually, each token has a length and a metadata number.
430 > * A token array might be used to annotate a string with metadata.
431 > * Use {@link TokenArrayBuilder} to efficiently create a token array.
432 > *
433 > * TODO: Make this class more efficient (e.g. by using a Int32Array).
434 > */
435 > export class TokenArray {
436 > public static fromLineTokens(lineTokens: LineTokens): TokenArray {
437 > const tokenInfo: TokenInfo[] = [];
438 > for (let i = 0; i < lineTokens.getCount(); i++) {
439 > tokenInfo.push(new TokenInfo(lineTokens.getEndOffset(i) - lineTokens.getStartOffset(i), lineTokens.getMetadata(i)));
440 > }
441 > return TokenArray.create(tokenInfo);
442 > }
443 >
444 > public static create(tokenInfo: TokenInfo[]): TokenArray {
445 return new TokenArray(tokenInfo);
446 }
447 > lineTokens.ts
448 > private constructor(
449 private readonly _tokenInfo: TokenInfo[]
450 ) { }
451 > lineTokens.ts
452 > public toLineTokens(lineContent: string, decoder: ILanguageIdCodec): LineTokens {
453 return LineTokens.createFromTextAndMetadata(this.map((r, t) => ({ text: r.substring(lineContent), metadata: t.metadata })), decoder);
454 }
455 > lineTokens.ts
456 > public forEach(cb: (range: OffsetRange, tokenInfo: TokenInfo) => void): void {
457 let lengthSum = 0;
458 for (const tokenInfo of this._tokenInfo) {
462 }
463 }
464 > lineTokens.ts
465 > public map<T>(cb: (range: OffsetRange, tokenInfo: TokenInfo) => T): T[] {
466 const result: T[] = [];
467 let lengthSum = 0;
473 return result;
474 }
475 > lineTokens.ts
476 > public slice(range: OffsetRange): TokenArray {
477 const result: TokenInfo[] = [];
478 let lengthSum = 0;
495 return TokenArray.create(result);
496 }
497 > lineTokens.ts
498 > public append(other: TokenArray): TokenArray {
499 const result: TokenInfo[] = this._tokenInfo.concat(other._tokenInfo);
500 return TokenArray.create(result);
501 }
502 > } lineTokens.ts
503 >
504 > export type ITokenMetadata = number;
505 >
506 > export class TokenInfo {
507 > constructor(
508 public readonly length: number,
509 public readonly metadata: ITokenMetadata
510 ) { }
511 > } lineTokens.ts
512 > /**
513 > * TODO: Make this class more efficient (e.g. by using a Int32Array).
514 > */
515 >
516 > export class TokenArrayBuilder {
517 private readonly _tokens: TokenInfo[] = [];
518 > lineTokens.ts
519 > public add(length: number, metadata: ITokenMetadata): void {
520 this._tokens.push(new TokenInfo(length, metadata));
521 }
522 > lineTokens.ts
523 > public build(): TokenArray {
524 return TokenArray.create(this._tokens);
525 }
526 > } lineTokens.ts
527

Direct hierarchy

Parents

Children