Hello, world!
", { jsx: true })`. ### JavaScript runtimes js-tokens should work in any JavaScript runtime that supports [Unicode property escapes]. ### Known errors Here are a couple of tricky cases: ```js // Case 1: switch (x) { case x: {}/a/g; case x: {}x
/g;
}
// Case 2:
label: {}/a/g;
label: {}x
/g;
// Case 3:
(function f() {}/a/g);
(function f() {}x
/g);
```
This is what they mean:
```js
// Case 1:
switch (x) {
case x:
{
}
/a/g;
case x:
{
}
x
/ g;
}
// Case 2:
label: {
}
/a/g;
label: {
}
x
/ g;
// Case 3:
(function f() {} / a / g);
(function f() {} < div > x < /div>/g);
```
But js-tokens thinks they mean:
```js
// Case 1:
switch (x) {
case x:
({} / a / g);
case x:
({} < div > x < /div>/g);
}
// Case 2:
label: ({} / a / g);
label: ({} < div > x < /div>/g);
// Case 3:
function f() {}
/a/g;
function f() {}
x
/ g;
```
In other words, js-tokens:
- Mis-identifies regex as division and JSX as comparison in case 1 and 2.
- Mis-identifies division as regex and comparison as JSX in case 3.
This happens because js-tokens looks at the previous token when deciding between regex and division or JSX and comparison. In these cases, the previous token is `}`, which either means “end of block” (→ regex/JSX) or “end of object literal” (→ division/comparison). How does js-tokens determine if the `}` belongs to a block or an object literal? By looking at the token before the matching `{`.
In case 1 and 2, that’s a `:`. A `:` _usually_ means that we have an object literal or ternary:
```js
let some = weird ? { value: {}/a/g } : {}/a/g;
```
But `:` is also used for `case` and labeled statements.
One idea is to look for `case` before the `:` as an exception to the rule, but it’s not so easy:
```js
switch (x) {
case weird ? true : {}/a/g: {}/a/g
}
```
The first `{}/a/g` is a division, while the second `{}/a/g` is an empty block followed by a regex. Both are preceded by a colon with a `case` on the same line, and it does not seem like you can distinguish between the two without implementing a parser.
Labeled statements are similarly difficult, since they are so similar to object literals:
```js
{
label: {}/a/g
}
({
key: {}/a/g
})
```
Finally, case 3 (`(function f() {}/a/g);`) is also difficult, because a `)` before a `{` means that the `{` is part of a _block,_ and blocks are _usually_ statements:
```js
if (x) {
}
/a/g;
function f() {}
/a/g;
```
But _function expressions_ are of course not statements. It’s difficult to tell an function _expression_ from a function _statement_ without parsing.
Luckily, none of these edge cases are likely to occur in real code.
## Performance
With [@babel/parser] for comparison. Node.js 18.1.0 on Ubuntu 20.04.
| Lines of code | Size | js-tokens@8.0.0 | @babel/parser@7.14.4 |
| ------------: | -------: | --------------: | -------------------: |
| ~100 | ~4.3 KiB | ~2 ms | ~17 ms |
| ~1 000 | ~39 KiB | ~5 ms | ~60 ms |
| ~10 000 | ~353 KiB | ~50 ms | ~273 ms |
| ~100 000 | ~4.7 MiB | ~385 ms | ~4.7 s |
| ~2 300 000 | ~132 MiB | ~10 s | failed (\*) |
(\*) Out of memory crash after ~9 minutes (even though I had increased the memory limit to 8 GiB).
See [benchmark.js] if you want to run benchmarks yourself.
[@babel/parser]: https://babeljs.io/docs/en/babel-parser
[additional syntax]: https://tc39.es/ecma262/#sec-additional-syntax
[annexb]: https://tc39.es/ecma262/#sec-additional-ecmascript-features-for-web-browsers
[benchmark.js]: benchmark.js
[divpunctuator]: https://tc39.es/ecma262/#prod-DivPunctuator
[ecmascript language: lexical grammar]: https://tc39.es/ecma262/#sec-ecmascript-language-lexical-grammar
[example.test.js]: test/example.test.js
[identifiername]: https://tc39.es/ecma262/#prod-IdentifierName
[identifierpart]: https://tc39.es/ecma262/#prod-IdentifierPart
[jsx specification]: https://facebook.github.io/jsx/
[lineterminatorsequence]: https://tc39.es/ecma262/#prod-LineTerminatorSequence
[multilinecomment]: https://tc39.es/ecma262/#prod-MultiLineComment
[nosubstitutiontemplate]: https://tc39.es/ecma262/#prod-NoSubstitutionTemplate
[numericliteral]: https://tc39.es/ecma262/#prod-annexB-NumericLiteral
[privateidentifier]: https://tc39.es/ecma262/#prod-PrivateIdentifier
[punctuator]: https://tc39.es/ecma262/#prod-Punctuator
[regularexpressionliteral]: https://tc39.es/ecma262/#prod-RegularExpressionLiteral
[rightbracepunctuator]: https://tc39.es/ecma262/#prod-RightBracePunctuator
[singlelinecomment]: https://tc39.es/ecma262/#prod-SingleLineComment
[stackoverflow-slash]: https://stackoverflow.com/a/27120110/2010616
[string.prototype.matchall]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/matchAll
[stringliteral]: https://tc39.es/ecma262/#prod-StringLiteral
[templatehead]: https://tc39.es/ecma262/#prod-TemplateHead
[templatemiddle]: https://tc39.es/ecma262/#prod-TemplateMiddle
[templatetail]: https://tc39.es/ecma262/#prod-TemplateTail
[test262-parser-tests]: https://github.com/tc39/test262-parser-tests
[tsx fixture]: test/fixtures/valid/jsx2.tsx
[typescript fixture]: test/fixtures/valid/typescript.module.ts
[unicode property escapes]: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Unicode_Property_Escapes
[whitespace]: https://tc39.es/ecma262/#prod-WhiteSpace
js-tokens-8.0.0/benchmark.js 0000664 0000000 0000000 00000011141 14240146155 0015724 0 ustar 00root root 0000000 0000000 "use strict";
const babelParser = require("@babel/parser");
const fastGlob = require("fast-glob");
const fs = require("fs");
const jsTokens = require("./build/index");
const usage = `
Usage: node benchmark.js [TOKENIZER] [NUM_FILES]
TOKENIZER:
- js-tokens (default)
- babel
NUM_FILES:
Positive integer.
# Run almost everything in node_modules through js-tokens:
node benchmark.js
# Run the 1000 shortest, non-empty JS files in node_modules through @babel/parser:
node benchmark.js babel 1000
`.trim();
function parseTokenizer(string) {
switch (string) {
case "babel":
case "js-tokens":
return string;
default:
throw new Error(`Expected "js-tokens" or "babel", but got: ${string}`);
}
}
function parseNumFiles(string) {
const number = Number(string);
if (Number.isFinite(number) && Number.isInteger(number) && number > 0) {
return number;
}
throw new Error(`Expected a positive integer, but got: ${string}`);
}
function parseArguments(argv) {
switch (argv.length) {
case 0:
return ["js-tokens", Infinity];
case 1:
return [parseTokenizer(argv[0]), Infinity];
case 2:
return [parseTokenizer(argv[0]), parseNumFiles(argv[1])];
default:
throw new Error(`Expected 0 to 2 arguments, but got: ${argv.length}`);
}
}
function read() {
return fastGlob
.sync("node_modules/**/*.js", {
ignore: ["node_modules/test262-parser-tests"],
})
.map((file) => [
file,
fs.readFileSync(file, "utf8").replace(/^#!.*\r?\n/, ""),
])
.filter(
([, content]) =>
/\S/.test(content) &&
(content.trim().includes("\n") || content.length > 100)
)
.sort(
([fileA, contentA], [fileB, contentB]) =>
contentA.length - contentB.length || fileA.localeCompare(fileB)
)
.map(([file, content]) => `/*${file.replace(/\*\//g, "* /")}*/ ${content}`);
}
function mean(array) {
return array.reduce((a, b) => a + b, 0) / array.length;
}
function median(array) {
const sorted = array.slice().sort((a, b) => a - b);
const half = sorted.length / 2;
return half === 0
? 0
: Number.isInteger(half)
? (sorted[half - 1] + sorted[half]) / 2
: sorted[Math.floor(half)];
}
function common(array) {
const map = new Map();
for (const i of array) {
map.set(i, (map.get(i) || 0) + 1);
}
return Array.from(map.entries())
.sort(([, a], [, b]) => b - a)
.slice(0, 4);
}
function nonEmptyLines(string) {
return string.split("\n").filter((line) => line.length > 0);
}
const KiB = 2 ** 10;
const MiB = 2 ** 20;
function fileSize(string) {
const byteLength = Buffer.byteLength(string, "utf8");
return byteLength < KiB
? `${byteLength} B`
: byteLength < MiB
? `${(byteLength / KiB).toFixed(1)} KiB`
: `${(byteLength / MiB).toFixed(1)} MiB`;
}
function run(argv) {
const [tokenizer, numFiles] = parseArguments(argv);
const allFiles = read();
const files = allFiles.slice(0, numFiles);
const allCode = allFiles.join("\n;");
const code = files.join("\n;");
const fileLengths = files.map((file) => file.length);
const lineLengths = nonEmptyLines(code).map((line) => line.length);
console.log("Total files:", allFiles.length);
console.log("Total chars:", allCode.length, fileSize(allCode));
console.log("Total lines:", nonEmptyLines(allCode).length);
if (code !== allCode) {
console.log("Used files: ", files.length);
console.log("Used chars: ", code.length, fileSize(code));
console.log("Used lines: ", nonEmptyLines(code).length);
}
console.log("File length mean: ", mean(fileLengths));
console.log("File length median:", median(fileLengths));
console.log("File length common:", common(fileLengths));
console.log("Line length mean: ", mean(lineLengths));
console.log("Line length median:", median(lineLengths));
console.log("Line length common:", common(lineLengths));
fs.writeFileSync("benchmark.input.js", code);
console.time(tokenizer);
if (tokenizer === "babel") {
try {
const { tokens } = babelParser.parse(code, {
tokens: true,
errorRecovery: true,
plugins: ["jsx", "flow"],
});
console.log("Number of tokens:", tokens.length);
} catch (error) {
console.error(error);
}
} else {
// `Array.from` causes an out-of-memory crash in Node.js 10.
const tokens = [];
for (const token of jsTokens(code, { jsx: true })) {
tokens.push(token);
}
console.log("Number of tokens:", tokens.length);
}
console.timeEnd(tokenizer);
}
try {
run(process.argv.slice(2));
} catch (error) {
console.error(error.message);
console.log();
console.log(usage);
process.exit(1);
}
js-tokens-8.0.0/build.js 0000664 0000000 0000000 00000002203 14240146155 0015070 0 ustar 00root root 0000000 0000000 "use strict";
const coffee = require("coffeescript");
const fs = require("fs");
const path = require("path");
const DIR = __dirname;
const BUILD = path.join(DIR, "build");
const READ_MORE = "**[➡️ Full readme](https://github.com/lydell/js-tokens/)**";
const FILES_TO_COPY = [
{ src: "LICENSE" },
{ src: "index.d.ts" },
{ src: "package-real.json", dest: "package.json" },
{
src: "README.md",
transform: (content) => content.replace(/^##[^]*/m, READ_MORE),
},
{
src: "index.coffee",
dest: "index.js",
transform: (content) =>
coffee
.compile(content, { bare: true })
.replace(/ {2}/g, "\t")
.replace(/\/\/ (?:Note:|https).*\n/g, "")
.replace(/\n\n/g, "\n")
.replace(/\{\s*(tag: "[^"]+")\s*\}/g, "{$1}"),
},
];
fs.rmSync(BUILD, { recursive: true, force: true });
fs.mkdirSync(BUILD);
for (const { src, dest = src, transform } of FILES_TO_COPY) {
if (transform) {
fs.writeFileSync(
path.join(BUILD, dest),
transform(fs.readFileSync(path.join(DIR, src), "utf8"))
);
} else {
fs.copyFileSync(path.join(DIR, src), path.join(BUILD, dest));
}
}
js-tokens-8.0.0/index.coffee 0000664 0000000 0000000 00000033237 14240146155 0015726 0 ustar 00root root 0000000 0000000 # Copyright 2014, 2015, 2016, 2017, 2018, 2019, 2020 Simon Lydell
# License: MIT.
# https://tc39.es/ecma262/#sec-lexical-grammar
# https://mathiasbynens.be/notes/javascript-identifiers
# https://github.com/tc39/proposal-regexp-unicode-property-escapes/#other-examples
# https://unicode.org/reports/tr31/#Backward_Compatibility
# https://stackoverflow.com/a/27120110/2010616
RegularExpressionLiteral = ///
/(?![ * / ])
(?:
\[
(?:
(?![ \] \\ ]).
|
\\.
)*
\]
|
(?![ / \] \\ ]).
|
\\.
)*
(
/
[ $ _ \u200C \u200D \p{ID_Continue} ]*
|
\\
)?
///yu
Punctuator = ///
-- | \+\+
|
=>
|
\.{3}
|
\??\. (?!\d)
|
(?:
&& | \|\| | \?\?
|
[ + \- % & | ^ ]
|
\*{1,2}
|
<{1,2} | >{1,3}
|
!=? | ={1,2}
|
/(?![ / * ])
)=?
|
[ ? ~ , : ; [ \] ( ) { } ]
///y
# Note: `\x23` is `#`. The escape is used since VSCode’s syntax highlighting breaks otherwise.
Identifier = ///
(\x23?)
(?=[ $ _ \p{ID_Start} \\ ])
(?:
[ $ _ \u200C \u200D \p{ID_Continue} ]
|
\\u[ \d a-f A-F ]{4}
|
\\u\{[ \d a-f A-F ]+\}
)+
///yu
StringLiteral = ///
([ ' " ])
(?:
(?! \1 )[^ \\ \n \r ]
|
\\(?: \r\n | [^] )
)*
(\1)?
///y
NumericLiteral = ///
(?:
0[xX][ \d a-f A-F ] (?: _? [ \d a-f A-F ] )*
|
0[oO][0-7] (?: _? [0-7] )*
|
0[bB][01] (?: _? [01] )*
)n?
|
0n
|
[1-9](?: _? \d )*n
|
(?:
(?:
0(?!\d)
|
0\d*[89]\d*
|
[1-9](?: _? \d )*
)
(?: \.(?: \d (?: _? \d )* )? )?
|
\.\d (?: _? \d )*
)
(?: [eE][+-]?\d (?: _? \d )* )?
|
0[0-7]+
///y
Template = ///
[ ` } ]
(?:
[^ ` \\ $ ]
|
\\[^]
|
\$(?!\{)
)*
( ` | \$\{ )?
///y
WhiteSpace = ///
[ \t \v \f \ufeff \p{Zs} ]+
///yu
LineTerminatorSequence = ///
\r?\n
|
[ \r \u2028 \u2029 ]
///y
MultiLineComment = ///
/\*
(?:
[^*]
|
\*(?!/)
)*
(\*/)?
///y
SingleLineComment = ///
//.*
///y
JSXPunctuator = ///
[ < > . : = { } ]
|
/(?![ / * ])
///y
JSXIdentifier = ///
[ $ _ \p{ID_Start} ]
[ $ _ \u200C \u200D \p{ID_Continue} - ]*
///yu
JSXString = ///
([ ' " ])
(?:
(?! \1 )[^]
)*
(\1)?
///y
JSXText = ///
[^ < > { } ]+
///y
TokensPrecedingExpression = ///
^(?:
[/+-]
|
\.{3}
|
\?(?:InterpolationIn(?:JSX|Template)|NoLineTerminatorHere|NonExpressionParenEnd|UnaryIncDec)
)?$
|
[ { } ( [ , ; < > = * % & | ^ ! ~ ? : ]$
///
TokensNotPrecedingObjectLiteral = ///
^(?:
=>
|
[ ; \] ) { } ]
|
else
|
\?(?:NoLineTerminatorHere|NonExpressionParenEnd)
)?$
///
KeywordsWithExpressionAfter = ///
^(?:await|case|default|delete|do|else|instanceof|new|return|throw|typeof|void|yield)$
///
KeywordsWithNoLineTerminatorAfter = ///
^(?:return|throw|yield)$
///
Newline = RegExp(LineTerminatorSequence.source)
module.exports = jsTokens = (input, {jsx = false} = {}) ->
{length} = input
lastIndex = 0
lastSignificantToken = ""
stack = [{tag: "JS"}]
braces = []
parenNesting = 0
postfixIncDec = false
while lastIndex < length
mode = stack[stack.length - 1]
switch mode.tag
when "JS", "JSNonExpressionParen", "InterpolationInTemplate", "InterpolationInJSX"
if input[lastIndex] == "/" && (
TokensPrecedingExpression.test(lastSignificantToken) ||
KeywordsWithExpressionAfter.test(lastSignificantToken)
)
RegularExpressionLiteral.lastIndex = lastIndex
if match = RegularExpressionLiteral.exec(input)
lastIndex = RegularExpressionLiteral.lastIndex
lastSignificantToken = match[0]
postfixIncDec = true
yield {
type: "RegularExpressionLiteral",
value: match[0],
closed: match[1] != undefined && match[1] != "\\",
}
continue
Punctuator.lastIndex = lastIndex
if match = Punctuator.exec(input)
punctuator = match[0]
nextLastIndex = Punctuator.lastIndex
nextLastSignificantToken = punctuator
switch punctuator
when "("
if lastSignificantToken == "?NonExpressionParenKeyword"
stack.push({tag: "JSNonExpressionParen", nesting: parenNesting})
parenNesting++
postfixIncDec = false
when ")"
parenNesting--
postfixIncDec = true
if mode.tag == "JSNonExpressionParen" && parenNesting == mode.nesting
stack.pop()
nextLastSignificantToken = "?NonExpressionParenEnd"
postfixIncDec = false
when "{"
Punctuator.lastIndex = 0
isExpression =
!TokensNotPrecedingObjectLiteral.test(lastSignificantToken) && (
TokensPrecedingExpression.test(lastSignificantToken) ||
KeywordsWithExpressionAfter.test(lastSignificantToken)
)
braces.push(isExpression)
postfixIncDec = false
when "}"
switch mode.tag
when "InterpolationInTemplate"
if braces.length == mode.nesting
Template.lastIndex = lastIndex
match = Template.exec(input)
lastIndex = Template.lastIndex
lastSignificantToken = match[0]
if match[1] == "${"
lastSignificantToken = "?InterpolationInTemplate"
postfixIncDec = false
yield {
type: "TemplateMiddle",
value: match[0],
}
else
stack.pop()
postfixIncDec = true
yield {
type: "TemplateTail",
value: match[0],
closed: match[1] == "`",
}
continue
when "InterpolationInJSX"
if braces.length == mode.nesting
stack.pop()
lastIndex += 1
lastSignificantToken = "}"
yield {
type: "JSXPunctuator",
value: "}"
}
continue
postfixIncDec = braces.pop()
nextLastSignificantToken =
if postfixIncDec then "?ExpressionBraceEnd" else "}"
when "]"
postfixIncDec = true
when "++", "--"
nextLastSignificantToken =
if postfixIncDec then "?PostfixIncDec" else "?UnaryIncDec"
when "<"
if jsx && (
TokensPrecedingExpression.test(lastSignificantToken) ||
KeywordsWithExpressionAfter.test(lastSignificantToken)
)
stack.push({tag: "JSXTag"})
lastIndex += 1
lastSignificantToken = "<"
yield {
type: "JSXPunctuator",
value: punctuator,
}
continue
postfixIncDec = false
else
postfixIncDec = false
lastIndex = nextLastIndex
lastSignificantToken = nextLastSignificantToken
yield {
type: "Punctuator",
value: punctuator,
}
continue
Identifier.lastIndex = lastIndex
if match = Identifier.exec(input)
lastIndex = Identifier.lastIndex
nextLastSignificantToken = match[0]
switch match[0]
when "for", "if", "while", "with"
if lastSignificantToken != "." && lastSignificantToken != "?."
nextLastSignificantToken = "?NonExpressionParenKeyword"
lastSignificantToken = nextLastSignificantToken
postfixIncDec = !KeywordsWithExpressionAfter.test(match[0])
yield {
type: if match[1] == "#" then "PrivateIdentifier" else "IdentifierName",
value: match[0],
}
continue
StringLiteral.lastIndex = lastIndex
if match = StringLiteral.exec(input)
lastIndex = StringLiteral.lastIndex
lastSignificantToken = match[0]
postfixIncDec = true
yield {
type: "StringLiteral",
value: match[0],
closed: match[2] != undefined,
}
continue
NumericLiteral.lastIndex = lastIndex
if match = NumericLiteral.exec(input)
lastIndex = NumericLiteral.lastIndex
lastSignificantToken = match[0]
postfixIncDec = true
yield {
type: "NumericLiteral",
value: match[0],
}
continue
Template.lastIndex = lastIndex
if match = Template.exec(input)
lastIndex = Template.lastIndex
lastSignificantToken = match[0]
if match[1] == "${"
lastSignificantToken = "?InterpolationInTemplate"
stack.push({tag: "InterpolationInTemplate", nesting: braces.length})
postfixIncDec = false
yield {
type: "TemplateHead",
value: match[0],
}
else
postfixIncDec = true
yield {
type: "NoSubstitutionTemplate",
value: match[0],
closed: match[1] == "`",
}
continue
when "JSXTag", "JSXTagEnd"
JSXPunctuator.lastIndex = lastIndex
if match = JSXPunctuator.exec(input)
lastIndex = JSXPunctuator.lastIndex
nextLastSignificantToken = match[0]
switch match[0]
when "<"
stack.push({tag: "JSXTag"})
when ">"
stack.pop()
if lastSignificantToken == "/" || mode.tag == "JSXTagEnd"
nextLastSignificantToken = "?JSX"
postfixIncDec = true
else
stack.push({tag: "JSXChildren"})
when "{"
stack.push({tag: "InterpolationInJSX", nesting: braces.length})
nextLastSignificantToken = "?InterpolationInJSX"
postfixIncDec = false
when "/"
if lastSignificantToken == "<"
stack.pop()
if stack[stack.length - 1].tag == "JSXChildren"
stack.pop()
stack.push({tag: "JSXTagEnd"})
lastSignificantToken = nextLastSignificantToken
yield {
type: "JSXPunctuator",
value: match[0],
}
continue
JSXIdentifier.lastIndex = lastIndex
if match = JSXIdentifier.exec(input)
lastIndex = JSXIdentifier.lastIndex
lastSignificantToken = match[0]
yield {
type: "JSXIdentifier",
value: match[0],
}
continue
JSXString.lastIndex = lastIndex
if match = JSXString.exec(input)
lastIndex = JSXString.lastIndex
lastSignificantToken = match[0]
yield {
type: "JSXString",
value: match[0],
closed: match[2] != undefined,
}
continue
when "JSXChildren"
JSXText.lastIndex = lastIndex
if match = JSXText.exec(input)
lastIndex = JSXText.lastIndex
lastSignificantToken = match[0]
yield {
type: "JSXText",
value: match[0],
}
continue
switch input[lastIndex]
when "<"
stack.push({tag: "JSXTag"})
lastIndex++
lastSignificantToken = "<"
yield {
type: "JSXPunctuator",
value: "<",
}
continue
when "{"
stack.push({tag: "InterpolationInJSX", nesting: braces.length})
lastIndex++
lastSignificantToken = "?InterpolationInJSX"
postfixIncDec = false
yield {
type: "JSXPunctuator",
value: "{",
}
continue
WhiteSpace.lastIndex = lastIndex
if match = WhiteSpace.exec(input)
lastIndex = WhiteSpace.lastIndex
yield {
type: "WhiteSpace",
value: match[0],
}
continue
LineTerminatorSequence.lastIndex = lastIndex
if match = LineTerminatorSequence.exec(input)
lastIndex = LineTerminatorSequence.lastIndex
postfixIncDec = false
if KeywordsWithNoLineTerminatorAfter.test(lastSignificantToken)
lastSignificantToken = "?NoLineTerminatorHere"
yield {
type: "LineTerminatorSequence",
value: match[0],
}
continue
MultiLineComment.lastIndex = lastIndex
if match = MultiLineComment.exec(input)
lastIndex = MultiLineComment.lastIndex
if Newline.test(match[0])
postfixIncDec = false
if KeywordsWithNoLineTerminatorAfter.test(lastSignificantToken)
lastSignificantToken = "?NoLineTerminatorHere"
yield {
type: "MultiLineComment",
value: match[0],
closed: match[1] != undefined,
}
continue
SingleLineComment.lastIndex = lastIndex
if match = SingleLineComment.exec(input)
lastIndex = SingleLineComment.lastIndex
postfixIncDec = false
yield {
type: "SingleLineComment",
value: match[0],
}
continue
firstCodePoint = String.fromCodePoint(input.codePointAt(lastIndex))
lastIndex += firstCodePoint.length
lastSignificantToken = firstCodePoint
postfixIncDec = false
yield {
type: if mode.tag.startsWith("JSX") then "JSXInvalid" else "Invalid",
value: firstCodePoint,
}
undefined
js-tokens-8.0.0/index.d.ts 0000664 0000000 0000000 00000002515 14240146155 0015342 0 ustar 00root root 0000000 0000000 export declare type Token =
| { type: "StringLiteral"; value: string; closed: boolean }
| { type: "NoSubstitutionTemplate"; value: string; closed: boolean }
| { type: "TemplateHead"; value: string }
| { type: "TemplateMiddle"; value: string }
| { type: "TemplateTail"; value: string; closed: boolean }
| { type: "RegularExpressionLiteral"; value: string; closed: boolean }
| { type: "MultiLineComment"; value: string; closed: boolean }
| { type: "SingleLineComment"; value: string }
| { type: "IdentifierName"; value: string }
| { type: "PrivateIdentifier"; value: string }
| { type: "NumericLiteral"; value: string }
| { type: "Punctuator"; value: string }
| { type: "WhiteSpace"; value: string }
| { type: "LineTerminatorSequence"; value: string }
| { type: "Invalid"; value: string };
export declare type JSXToken =
| { type: "JSXString"; value: string; closed: boolean }
| { type: "JSXText"; value: string }
| { type: "JSXIdentifier"; value: string }
| { type: "JSXPunctuator"; value: string }
| { type: "JSXInvalid"; value: string };
export default function jsTokens(
input: string,
options: { jsx: true }
): Iterable![](\"https://example.com/logo.png\"){kind=logo}
} */",">",
"\n ","<","//c1","\n",
" ","/*c2*/"," ","xml",":","nav-item","//c2","\n",
" ","/*c3*/",">",
"\n line one\n next line\n ",
"<","//c4","\n",
" ","/*c5*/","//*c6*/","\n",
" ","/*c7*/","/"," ","/*c8*/","\n",
" ","/*c9*/","xml",":","nav-item","//c10","\n",
" ","/*c11*/",">",
"\n\n ","<","div",">",
"\n ","<","p",">",
"\n ","{","/* comment */","}",
"\n ","<","a"," ","href","=","{","`https://${","domain","}.com/${","<","path",">","{","`${","{","}","<","x",">","y","<","/x/g","}`","}","<","/","path",">",".","toString","(",")","}`","}","\n",
" ","alt","/**/","=","/**/","<","Alt"," ","text","=","<","Text","/",">","/",">","\n",
" ",">",
"\n Some","{","' '","}","link\n ",
"<","/","a",">",
"\n ","{","// comment","\n",
" ","}",
"\n ","<","/","p",">",
"\n ","<","/","div",">",
"\n","<","/","Global",".","Nav",".","Component",">",";","\n"
]
js-tokens-8.0.0/test/fixtures/valid/jsx1.jsx 0000664 0000000 0000000 00000001173 14240146155 0021002 0 ustar 00root root 0000000 0000000 
}
/* logo={
} */>
line one
next line
{/* comment */}
{`${{}