tfIdf.ts ×8

Frontier kind: Code frontier

unlabeled · c_24ae3fdea353

9 tests · 4553 LOC · 23 files · introduces 0 tests · 48 LOC · 1 file

Introduces — evidence that enters the hierarchy at this concept

Code: 8 ranges48 lines · 1 files
Tests: 0 tests

Contains — complete concept membership

All code (extent): 627 ranges4553 lines · 23 files · Browse complete extent
All tests (intent): 9 testsBrowse complete intent

Neighbourhood graph

The orange circle is the focus. Violet and green circles are every ancestor and descendant, broader and narrower, at any distance; blue squares and pink diamonds are the introduced files and exact introduced tests of every visible concept, not only the focus's. Arrows point from broader to narrower concepts and bridge only concepts omitted from this view. Undirected links show source or test introduction. Concept and file size follows LOC; exact test nodes use test-count units.

Focus concept
Ancestors (broader)
Descendants (narrower)
Introduced files
Introduced tests

Introduced files, introduced tests, and structurally relevant concept specialization

In the embedded map, ordinary wheel input scrolls the page; use the visible controls to zoom and drag to pan. Open the full-screen map for canvas navigation: wheel pans, Ctrl/Command plus wheel zooms, and arrow keys pan when this region is focused. On touch screens, open the full-screen map to pan or pinch. If JavaScript or WebGL is unavailable, use the native relationship evidence on this page.

Native relationship evidence

Every exact file and test below is linked only from the concept that introduces it.

tfIdf.ts ×8 — Focused concept Introduced files:
- src/vs/base/common/tfIdf.ts
tfIdf.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/base/common/tfIdf.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/tfIdf.test|title=TF-IDF Calculator Should match on camelCase words|occurrence=1
tfIdf.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/base/common/tfIdf.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/tfIdf.test|title=TF-IDF Calculator Should not match document after delete|occurrence=1
tfIdf.ts ×2 — Ancestor (broader) Introduced files:
- src/vs/base/common/tfIdf.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/tfIdf.test|title=TF-IDF Calculator Should ignore case and punctuation|occurrence=1
tfIdf.test|title=TF-IDF Calculator Should weigh less common term higher|occurrence=1, tfIdf.test|title=TF-IDF Calculator Should weigh chunks with less common terms higher|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/tfIdf.test|title=TF-IDF Calculator Should weigh chunks with less common terms higher|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/tfIdf.test|title=TF-IDF Calculator Should weigh less common term higher|occurrence=1
tfIdf.ts ×1 — Ancestor (broader) Introduced files:
- src/vs/base/common/tfIdf.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/tfIdf.test|title=TF-IDF Calculator Should return document with more matches first|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/tfIdf.test|title=TF-IDF Calculator Should return scores for document with exact match|occurrence=1
tfIdf.ts ×5 — Ancestor (broader) Introduced files:
- src/vs/base/common/tfIdf.ts
Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/tfIdf.test|title=TF-IDF Calculator Should return document with more matches first when term appears in all documents|occurrence=1
tfIdf.test|title=TF-IDF Calculator Should return no scores for term not in document|occurrence=1 — Ancestor (broader) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/tfIdf.test|title=TF-IDF Calculator Should return no scores for term not in document|occurrence=1
tfIdf.ts ×8 — Descendant (narrower) Introduced files:
- src/vs/base/common/tfIdf.ts
cancellation.test|title=CancellationToken None|occurrence=1 — Descendant (narrower) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/cancellation.test|title=CancellationToken None|occurrence=1
cancellation.ts ×18 — Descendant (narrower) Introduced files:
- src/vs/base/common/cancellation.ts
event.ts ×93 — Descendant (narrower) Introduced files:
- src/vs/base/common/event.ts
- src/vs/base/common/stopwatch.ts
uri.test|title=URI URI#file, win-speciale|occurrence=1, charCode.test|title=CharCode has good values|occurrence=1, +2 — Descendant (narrower) Introduced tests:
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/charCode.test|title=CharCode has good values|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/path.test|title=Paths (Node Implementation) path|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/uri.test|title=URI File paths containing apostrophes break URI parsing and cannot be opened #276075|occurrence=1
- mocha:v1|namespace=vscode@05c208e9e28d8c1c723fa08f85e2b7a96092e8e5|file=vs/base/test/common/uri.test|title=URI URI#file, win-speciale|occurrence=1
utils.ts ×3 — Descendant (narrower) Introduced files:
- src/vs/base/test/common/utils.ts
linkedList.ts ×13 — Descendant (narrower) Introduced files:
- src/vs/base/common/linkedList.ts
lifecycle.ts ×2 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×4 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×6 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
lifecycle.ts ×1 — Descendant (narrower) Introduced files:
- src/vs/base/common/lifecycle.ts
map.ts ×97 — Descendant (narrower) Introduced files:
- src/vs/base/common/arrays.ts
- src/vs/base/common/arraysFind.ts
- src/vs/base/common/assert.ts
- src/vs/base/common/charCode.ts
- src/vs/base/common/collections.ts
- src/vs/base/common/errors.ts
- src/vs/base/common/functional.ts
- src/vs/base/common/iterator.ts
- src/vs/base/common/lifecycle.ts
- src/vs/base/common/map.ts
- src/vs/base/common/marshallingIds.ts
- src/vs/base/common/path.ts
- src/vs/base/common/platform.ts
- src/vs/base/common/process.ts
- src/vs/base/common/types.ts
- src/vs/base/common/uri.ts
- src/vs/base/test/common/utils.ts
- src/vs/nls.ts

Introduced tests

Every collected test enters the hierarchy at exactly one concept.

No tests are introduced at this concept. Its intent tests are introduced by other concepts.

Introduced code

Every collected source range enters the hierarchy at exactly one concept.

1 file ranked by introduced lines: 48 introduced LOC across 8 ranges. Expand a file to inspect source; the > gutter marks introduced lines.

src/vs/base/common/tfIdf.ts 48 introduced LOC · 8 ranges

Open complete file

  57
  
  		// For each document, generate one score
  
  58
  
  		for (const [key, doc] of this.documents) {
  
  59
  >
  			if (token.isCancellationRequested) {
  
tfIdf.ts

  60
  
  				return [];
  
  61
  
  			}
  
  62
  >
  
tfIdf.ts

  63
  >
  			for (const chunk of doc.chunks) {
  
  64
  >
  				const score = this.computeSimilarityScore(chunk, embedding, idfCache);
  
  65
  >
  				if (score > 0) {
  
  66
  
  					scores.push({ key, score });
  
  67
  
  				}
  
  68
  >
  			}
  
tfIdf.ts

  69
  >
  		}
  
  70
  
  71
  
  		return scores;

  113
  
  114
  
  	updateDocuments(documents: ReadonlyArray<TfIdfDocument>): this {
  
  115
  >
  		for (const { key } of documents) {
  
tfIdf.ts

  116
  >
  			this.deleteDocument(key);
  
  117
  >
  		}
  
  118
  >
  
  119
  >
  		for (const doc of documents) {
  
  120
  >
  			const chunks: Array<{ text: string; tf: TermFrequencies }> = [];
  
  121
  >
  			for (const text of doc.textChunks) {
  
  122
  >
  				// TODO: See if we can compute the tf lazily
  
  123
  >
  				// The challenge is that we need to also update the `chunkOccurrences`
  
  124
  >
  				// and all of those updates need to get flushed before the real TF-IDF of
  
  125
  >
  				// anything is computed.
  
  126
  >
  				const tf = TfIdfCalculator.termFrequencies(text);
  
  127
  >
  
  128
  >
  				// Update occurrences list
  
  129
  >
  				for (const term of tf.keys()) {
  
  130
  >
  					this.chunkOccurrences.set(term, (this.chunkOccurrences.get(term) ?? 0) + 1);
  
  131
  >
  				}
  
  132
  >
  
  133
  >
  				chunks.push({ text, tf });
  
  134
  >
  			}
  
  135
  >
  
  136
  >
  			this.chunkCount += chunks.length;
  
  137
  >
  			this.documents.set(doc.key, { chunks });
  
  138
  >
  		}
  
  139
  >
  		return this;
  
  140
  >
  	}
  
  141
  
  142
  
  	deleteDocument(key: string) {
  
  143
  >
  		const doc = this.documents.get(key);
  
tfIdf.ts

  144
  >
  		if (!doc) {
  
  145
  >
  			return;
  
  146
  >
  		}
  
  147
  
  148
  
  		this.documents.delete(key);

  163
  
  			}
  
  164
  
  		}
  
  165
  >
  	}
  
tfIdf.ts

  166
  
  167
  
  	private computeSimilarityScore(chunk: DocumentChunkEntry, queryEmbedding: SparseEmbedding, idfCache: Map<string, number>): number {
  
  168
  >
  		// Compute the dot product between the chunk's embedding and the query embedding
  
tfIdf.ts

  169
  >
  
  170
  >
  		// Note that the chunk embedding is computed lazily on a per-term basis.
  
  171
  >
  		// This lets us skip a large number of calculations because the majority
  
  172
  >
  		// of chunks do not share any terms with the query.
  
  173
  >
  
  174
  >
  		let sum = 0;
  
  175
  >
  		for (const [term, termTfidf] of Object.entries(queryEmbedding)) {
  
  176
  
  			const chunkTf = chunk.tf.get(term);
  
  177
  
  			if (!chunkTf) {

  189
  
  			sum += chunkTfidf * termTfidf;
  
  190
  
  		}
  
  191
  >
  		return sum;
  
tfIdf.ts

  192
  >
  	}
  
  193
  
  194
  
  	private computeEmbedding(input: string): SparseEmbedding {

tfIdf.ts ×8

Introduces — evidence that enters the hierarchy at this concept

Contains — complete concept membership

Neighbourhood graph

Introduced tests

Introduced code

Direct hierarchy

Parents

Children