mmark-1.3-git849458a/000077500000000000000000000000001264073172300141215ustar00rootroot00000000000000mmark-1.3-git849458a/.gitignore000066400000000000000000000000601264073172300161050ustar00rootroot00000000000000tags mmark/mmark ietf/eppext-keyrelay.pdc *.swp mmark-1.3-git849458a/.rel.sh000077500000000000000000000013471264073172300153250ustar00rootroot00000000000000#!/bin/bash FILES="mmark/mmark CONVERSION_RFC7328.md mmark2rfc.md README.md skel.md misc rfc" ( cd mmark ; make clean; make ) VERSION=$(mmark/mmark -version) # Linux export GOOS=linux GOARCH=amd64 ( cd mmark ; make clean; make ) dir=$(mktemp -d) mkdir ${dir}/mmark trap "rm -rf $dir" EXIT cp -r $FILES $dir/mmark ( cd $dir; \ tar --verbose --create --bzip2 --file /tmp/mmark-v$VERSION-$GOOS-$GOARCH.tar.bz2 mmark ) # Darwin export GOOS=darwin GOARCH=amd64 ( cd mmark ; make clean; make ) dir=$(mktemp -d) mkdir ${dir}/mmark trap "rm -rf $dir" EXIT cp -r $FILES $dir/mmark ( cd $dir; \ zip -r /tmp/mmark-v$VERSION-$GOOS-$GOARCH.zip mmark ) ls /tmp/mmark-v$VERSION-linux-$GOARCH.tar.bz2 ls /tmp/mmark-v$VERSION-darwin-$GOARCH.zip mmark-1.3-git849458a/.travis.yml000066400000000000000000000002371264073172300162340ustar00rootroot00000000000000language: go go: - 1.4 - 1.5 os: - linux - osx install: - go get -d -t -v ./... - go build -v ./... script: - go test -v ./... mmark-1.3-git849458a/CONVERSION_RFC7328.md000066400000000000000000000052641264073172300171550ustar00rootroot00000000000000# Converting From RFC 7328 Syntax Mmark can not directly parse an RFC 7328 style document, but pandoc can, and pandoc can output a document that *can* be parsed by mmark. The following (long-ish) commandline allows pandoc to parse the document and output something mmark can grok (main use here is to convert tables to the mmark table format): cat YOURFILE.md | \ pandoc --atx-headers -f markdown_phpextra+simple_tables+multiline_tables+grid_tables+superscript \ -t markdown_phpextra+superscript | \ sed 's|\\^\[|\^\[|' > YOURFILE_mmark.md This should deal with most of the constructs used in your original pandoc2rfc file. But be aware of the following limitations: * Indices (RFC 7328 Section 6.4), are detected *and* parsed. * Captions and anchors (Section 6.3) are detected *but not* parsed, instead a warning is given that the text should be reworked. * Makes all RFC references normative. * Abstract needs to be moved to a `.# Abstract` section. This leaves the title block, i.e. the `template.xml` from pandoc2rfc, which should be converted to a TOML titleblock, use `mmark -toml template.xml` for this. It should output a(n) (in)correct TOML title block that can be used as a starting point. ## Example To take a live example, consider [this draft middle piece](https://raw.githubusercontent.com/miekg/denialid/master/middle.mkd), which later became [RFC 7129](https://tools.ietf.org/html/rfc7129): % curl https://raw.githubusercontent.com/miekg/denialid/master/middle.mkd | \ pandoc --atx-headers \ -f markdown_phpextra+simple_tables+multiline_tables+grid_tables+superscript \ -t markdown_phpextra+superscript | \ sed 's|\\^\[|\^\[|' | ./mmark/mmark -rfc7328 -xml2 With warnings about the captions and anchors for figures: mmark: rfc 7328 style anchor seen: consider adding '{#fig:the-unsigned}' IAL before the figure/table mmark: rfc 7328 style caption seen: consider adding 'Figure: The unsigned "example.org" zone.' or 'Table: The unsigned "example.org" zone.' after the figure/table mmark: rfc 7328 style anchor seen: consider adding '{#fig:the-nsec-r}' IAL before the figure/table mmark: rfc 7328 style caption seen: consider adding 'Figure: The NSEC records of "example.org". The arrows represent NSEC records, starting from the apex.' or 'Table: The NSEC records of "example.org". The arrows represent NSEC records, starting from the apex.' after the figure/table .... ## Why Convert? If you want to submit your document in XML2RFC v3 (unlikely right now), or just want to make use of the speed, convenience (everything in one document) and extra features of mmark. If you are happy with pandoc2rfc then there is no need to convert. mmark-1.3-git849458a/COPYRIGHT000066400000000000000000000003221264073172300154110ustar00rootroot00000000000000Mmark Markdown Processor Available at http://github.com/miekg/mmark Copyright © 2014 Miek Gieben . Copyright © 2011 Russ Ross . Distributed under the Simplified BSD License. mmark-1.3-git849458a/LICENSE000066400000000000000000000026421264073172300151320ustar00rootroot00000000000000Mmark is distributed under the Simplified BSD License: > Copyright © 2014 Miek Gieben > Copyright © 2011 Russ Ross > All rights reserved. > > Redistribution and use in source and binary forms, with or without > modification, are permitted provided that the following conditions > are met: > > 1. Redistributions of source code must retain the above copyright > notice, this list of conditions and the following disclaimer. > > 2. Redistributions in binary form must reproduce the above > copyright notice, this list of conditions and the following > disclaimer in the documentation and/or other materials provided with > the distribution. > > THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS > "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT > LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS > FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE > COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, > INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, > BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; > LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER > CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT > LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN > ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE > POSSIBILITY OF SUCH DAMAGE. mmark-1.3-git849458a/Makefile000066400000000000000000000007311264073172300155620ustar00rootroot00000000000000MMARK:=./mmark/mmark -xml2 -page MMARK3:=./mmark/mmark -xml -page objects := README.md.txt mmark2rfc.md.txt %.md.txt: %.md $(MMARK) $< > $<.xml xml2rfc --text $<.xml && rm $<.xml all: mmark/mmark $(objects) mmark/mmark: ( cd mmark; make ) mmark2rfc.md.3.xml: mmark2rfc.md mmark/mmark $(MMARK3) $< > $<.3.xml .PHONY: clean clean: rm -f *.md.txt *md.[23].xml .PHONY: validate validate: mmark2rfc.md.3.xml xmllint --xinclude $< | jing -c xml2rfcv3.rnc /dev/stdin mmark-1.3-git849458a/README.md000066400000000000000000000401561264073172300154060ustar00rootroot00000000000000%%%%%%%%%%%%%%%%%%%%%%%%%%% Title = "Using mmark to create I-Ds, RFCs and books" abbrev = "mmark2rfc" category = "info" docName = "draft-gieben-mmark2rfc-00" [pi] private = "yes" footer = "" header = "mmark" date = 2015-10-10T00:00:00Z [[author]] initials="R." surname="Gieben" fullname="R. (Miek) Gieben" %%%%%%%%%%%%%%%%%%%%%%%%%%% .# Abstract ![mmark logo](logo/mmark120px.png) This document describes an markdown variant called mmark [@?mmark-ref]. Write RFCs using markdown. Mmark (written in Go) provides an advanced markdown dialect that processes a single file to produce internet-drafts in XML format. Internet-drafts written in mmark can produce XML2RFC v2 *and* XML2RFC v3 output. {mainmatter} # Mmark Mmark is a fork of blackfriday which is a [Markdown][1] processor implemented in [Go][2]. It supports a number of extensions, inspired by Leanpub, kramdown and Asciidoc, that allows for large documents to be written. It is specifically designed to write internet drafts (I-Ds) and RFCs for the IETF. With mmark you can create a single file that serves as input into the XML2RFC processor. It also allows for writing large documents such as technical books, like my [Learning Go book](https://github.com/miekg/learninggo). Sample text output of this book (when rendered as an I-D) can [be found here](https://gist.githubusercontent.com/miekg/0251f3e28652fa603a51/raw/7e0a7028506f7d2948e4ad3091f533711bf5f2a4/learninggo.txt). It is not perfect due to limitations in xml2rfc version 2. Fully rendered HTML version [can be found here](http://miek.nl/go). See `mmark2rfc.md` as an example I-D for how to use mmark. Or see the rfc/ subdirectory with some April 1st RFC that have been typeset in mmark. It can currently output HTML5, XML2RFC v2 and XML2RFC v3 XML. Other output engines could be added. It adds the following syntax elements to [black friday](https://github.com/russross/blackfriday/blob/master/README.md): * Definition lists. * More enumerated lists. * Table and codeblock captions. * Table footer. * Subfigures. * Quote attribution. * Including other files. * [TOML][3] titleblock. * Inline Attribute Lists. * Indices. * Citations. * Abstract/Preface/Notes sections. * Parts. * Asides. * Main-, middle- and backmatter divisions. * Math support. * Example lists. * HTML Comment parsing. * BCP14 (RFC2119) keyword detection. * Include raw XML references. * Abbreviations. * Super- and subscript. * Callouts in code blocks. Mmark is forked from blackfriday which started out as a translation from C of [upskirt][4]. A simular effort is [kramdown-rfc2629](https://github.com/cabo/kramdown-rfc2629) from Carsten Bormann. There is no pretty printed output if you need that pipe the output through `xmllint --format -`. ## Usage In the mmark subdirectory you can build the mmark tool: % cd mmark % go build % ./mmark -h Mmark Markdown Processor v1.0 ... To output v2 xml just give it a markdown file and: % ./mmark/mmark -xml2 -page mmark2rfc.md Making a draft in text form: % ./mmark/mmark -xml2 -page mmark2rfc.md > x.xml \ && xml2rfc --text x.xml \ && rm x.xml && mv x.txt mmark2rfc.txt Outputing v3 xml is done with the `-xml` switch. There is not yet a processor for this XML, but you should be able to validate the resulting XML against the schema from the XML2RFC v3 draft. I'm trying to stay current with the latest draft for the V3 spec: # Extensions In addition to the standard markdown syntax, this package implements the following extensions: * **Intra-word emphasis supression**. The `_` character is commonly used inside words when discussing code, so having markdown interpret it as an emphasis command is usually the wrong thing. Blackfriday lets you treat all emphasis markers as normal characters when they occur inside a word. * **Tables**. Tables can be created by drawing them in the input using a simple syntax: ``` Name | Age --------|-----: Bob | 27 Alice | 23 ``` Tables can also have a footer, use equal signs instead of dashes for the separator. If there are multiple footer lines, the first one is used as a starting point for the table footer. ``` Name | Age --------|-----: Bob | 27 Alice | 23 ======= | ==== Charlie | 4 ``` If a table is started with a *block table header*, which starts with a pipe or plus sign and a minimum of three dashes, it is a **Block Table**. A block table may include block level elements in each (body) cell. If we want to start a new cell reuse the block table header syntax. In the exampe below we include a list in one of the cells. ``` |+-----------------------------------------------| | Default aligned |Left aligned| Center aligned | |-----------------|:-----------|:---------------:| | First body part |Second cell | Third cell | | Second line |foo | **strong** | | Third line |quux | baz | |------------------------------------------------| | Second body | | 1. Item2 | | 2 line | | 2. Item2 | |================================================| | Footer row | | | |-----------------+------------+-----------------| ``` Note that the header and footer can't contain block level elements. Row spanning is supported as well, by using the [multiple pipe syntax](http://bywordapp.com/markdown/guide.html#section-mmd-tables-colspanning). * **Subfigure**. Fenced code blocks and indented code block can be grouped into a single figure containing both (or more) elements. Use the special quote prefix `F>` for this. * **Fenced code blocks**. In addition to the normal 4-space indentation to mark code blocks, you can explicitly mark them and supply a language (to make syntax highlighting simple). Just mark it like this: ``` go func getTrue() bool { return true } ``` You can use 3 or more backticks to mark the beginning of the block, and the same number to mark the end of the block. * **Autolinking**. Mmark can find URLs that have not been explicitly marked as links and turn them into links. * **Strikethrough**. Use two tildes to mark text that should be crossed out. * **Short References**. Internal references use the syntax `[](#id)`, usually the need for the title within the brackets is not needed, so mmark has the shorter syntax `(#id)` to cross reference in the document. * **Hard line breaks**. With this extension enabled newlines in the input translate into line breaks in the output. This is activate by using two trailing spaces before a new line. Another way to get a hard line break is to escape the newline with a \. And yet another another way to do this is to use 2 backslashes it the end of the line. * **Includes**, support including files with `{{filename}}` syntax. This is only done when include is started at the beginning of a line. * **Code Block Includes**, use the syntax `<{{code/hello.c}}[address]`, where address is the syntax described in , the OMIT keyword in the code also works. So including a code snippet will work like so: <{{test.go}}[/START OMIT/,/END OMIT/] where `test.go` looks like this: ``` go tedious_code = boring_function() // START OMIT interesting_code = fascinating_function() // END OMIT ``` To aid in including HTML or XML framents, where the `OMIT` key words is probably embedded in comments, lines which end in `OMIT -->` are also excluded. Of course the captioning works here as well: <{{test.go}}[/START OMIT/,/END OMIT/] Figure: A sample program. The address may be omitted: `<{{test.go}}` is legal as well. Note that the special `prefix` attribute can be set in an IAL and it will be used to prefix each line with the value of `prefix`. {prefix="S"} <{{test.go}} Will cause `test.go` to be included with each line being prefixed with `S`. * **Indices**, using `(((item, subitem)))` syntax. To make `item` primary, use an `!`: `(((!item, subitem)))`. Just `(((item)))` is allowed as well. * **Citations**, using the citation syntax from pandoc `[@RFC2535 p. 23]`, the citation can either be informative (default) or normative, this can be indicated by using the `?` or `!` modifer: `[@!RFC2535]`. Use `[-@RFC1000]` to add the cication to the references, but suppress the output in the document. The highest modifier seen determines the final type, i.e. once a citation is declared normative it will stay normative, but informative will be upgraded to normative. If you reference an RFC or I-D the reference will be contructed automatically. For I-Ds you may need to add a draft sequence number, which can be done as such: `[@?I-D.blah#06]`. If you have other references you can include the raw XML in the document (before the `{backmatter}`). Also see **XML references**. If you reference an I-D without a sequence number it will create a reference to the *last* I-D in citation index. Once a citation has been defined (i.e. the reference anchor is known to mmark) you can use `@RFC2535` is a shortcut for the citation. * **Captions**, table and figure/code block captions. For tables add the string `Table: caption text` after the table, this will be rendered as an caption. For code blocks you'll need to use `Figure: ` Name | Age --------|-----: Bob | 27 Alice | 23 Table: This is a table. Or for a code block: ``` go func getTrue() bool { return true } ``` Figure: Look! A Go function. * **Quote attribution**, after a blockquote you can optionally use `Quote: http://example.org -- John Doe`, where the quote will be attributed to John Doe, pointing to the URL: > Ability is nothing without opportunity. Quote: http://example.com -- Napoleon Bonaparte * **Abstracts**, use the special header `.# Abstract`. Note that the header name, when lowercased, must match 'abstract'. * **Notes**, use the special header `#. Title`, if the Title does *not* match (lowercase) abstract it will be a note. * **Asides**, any paragraph prefixed with `A>` . * **Subfigures**, any paraphgraph prefix with `F>` will wrap all images and code in a single figure. * **{frontmatter}/{mainmatter}/{backmatter}** Create useful divisions in your document. * **IAL**, kramdown's Inline Attribute List syntax, but took the CommonMark proposal, thus without the colon after the brace `{#id .class key=value key="value"}`. IALs are used for the following (block) elements: * Table * Code Block * Fenced Code Block * List (any type) * Section Header * Image * Quote * ... * **Definitition lists**, the markdown extra syntax. Apple : Pomaceous fruit of plants of the genus Malus in the family Rosaceae. Orange : The fruit of an evergreen tree of the genus Citrus. * **Enumerated lists**, roman, uppercase roman and normal letters can be used to start lists. Note that you'll need two space after the list counter: a. Item2 b. Item2 * **TOML TitleBlock**, add an extended title block prefixed with `%` in TOML. See the examples RFC in the rfc/ subdirectory. An alternate way of typsetting the TOML title block is wrapping it between two lines consisting out of 3 or more `%%%`s. Added benefit of wrapping it in `%%%`-lines is that the block may be indented (TOML disregards whitespace) 4 spaces, making it look like code in other markdown renderers (think Github). * **Unique anchors**, make anchors unique by adding sequence numbers (-1, -2, etc.) to them. All numeric section get an anchor prefixed with `section-`. * **Example lists**, a list that is started with `(@good)` is subsequently numbered i throughout the document. First use is rendered `(1)`, the second one `(2)` and so on. You can reference the last item of the list with `(@good)`. * **HTML comments** An HTML comment in the form of `` is detected and will be converted to a `cref` with the `source` attribute set to "Miek Gieben" and the comment text set to "really". * **XML references** Any XML reference fragment included *before* the back matter, can be used as a citation reference. * **BCP 14** If a RFC 2119 word is found enclosed in `**` it will be rendered as an `` element: `**MUST**` becomes `MUST`. * **Abbreviations**: See , any text defined by: *[HTML]: Hyper Text Markup Language Allows you to use HTML in the document and it will be expanded to `HTML`. If you need text that looks like an abbreviation, but isn't, escape the colon: *[HTML]\: HyperTextMarkupLanguage * **Super and subscripts**, for superscripts use '^' and for subscripts use '~'. For example: H~2~O is a liquid. 2^10^ is 1024. Inside a sub/superscript you must escape spaces. Thus, if you want the letter P with 'a cat' in subscripts, use `P~a\ cat~`, not `P~a cat~`. * **Parts**, use the special part header `-#` to start a new part. This follows the header syntax, so `-# Part {#part1}` is a valid part header. * **Math support**, use `$$` as the delimiter. If the math is part of a paragraph it will be displayed inline, if the entire paragraph consists out of math it considered display math. No attempt is made to parse what is between the `$$`. * **Callouts**, in codeblocks you can use `` to create a callout, later you can reference it: Code <1> More <1> Not a callout \<3> As you can see in <1> but not in \<1>. There is no <3>. You can escape a callout with a backslash. The backslash will be removed in the output (both in sourcecode and text). The callout identifiers will be remembered until the next code block. The above would render as: Code <1> Code <2> Not a callout <3> As you can see in (1, 2) but not in <1>. There is no <3>. Note that callouts are only detected with the IAL `{callout="yes"}` or any other non-empty value is defined before the code block. Now, you don't usualy want to globber your sourcecode with callouts as this will lead to code that does not compile. To fix this the callout needs to be placed in a comment, but then your source show useless empty comments. To fix this mmark can optionally detect (and remove!) the comment and the callout, leaving your example pristine. This can be enabled by setting `{callout="//"}` for instance. The allowed comment patterns are `//`, `#` and `;`. [1]: http://daringfireball.net/projects/markdown/ "Markdown" [2]: http://golang.org/ "Go Language" [3]: https://github.com/toml-lang/toml "TOML" [4]: http://github.com/tanoku/upskirt "Upskirt" [5]: http://github.com/russross/blackfriday "Blackfriday" Mmark: a powerful markdown processor in Go geared for the IETF Atoom Inc.
miek@miek.nl
 {backmatter} mmark-1.3-git849458a/block.go000066400000000000000000001406061264073172300155510ustar00rootroot00000000000000// Functions to parse block-level elements. package mmark import ( "bytes" "strconv" "unicode" ) // Parse block-level data. // Note: this function and many that it calls assume that // the input buffer ends with a newline. func (p *parser) block(out *bytes.Buffer, data []byte) { if len(data) == 0 || data[len(data)-1] != '\n' { panic("mmark: block input is missing terminating newline") } // this is called recursively: enforce a maximum depth if p.nesting >= p.maxNesting { return } p.nesting++ // parse out one block-level construct at a time for len(data) > 0 { // IAL // // {.class #id key=value} if data[0] == '{' { if j := p.isInlineAttr(data); j > 0 { data = data[j:] continue } } // part header: // // -# Part if p.flags&EXTENSION_PARTS != 0 { if p.isPartHeader(data) { data = data[p.partHeader(out, data):] continue } } // prefixed header: // // # Header 1 // ## Header 2 // ... // ###### Header 6 if p.isPrefixHeader(data) { data = data[p.prefixHeader(out, data):] continue } // special header: // // .# Abstract // .# Preface if p.isSpecialHeader(data) { if i := p.specialHeader(out, data); i > 0 { data = data[i:] continue } } // block of preformatted HTML: // //
// ... //
if data[0] == '<' { if i := p.html(out, data, true); i > 0 { data = data[i:] continue } } // title block in TOML // // % stuff = "foo" // % port = 1024 if p.flags&EXTENSION_TITLEBLOCK_TOML != 0 && len(data) > 2 { // only one % at the left if data[0] == '%' && data[1] != '%' { if i := p.titleBlock(out, data, true); i > 0 { data = data[i:] continue } } } // title block in TOML, second way to typeset // // %%% // stuff = "foot" // port = 1024 // %%% if p.flags&EXTENSION_TITLEBLOCK_TOML != 0 && len(data) > 3 { if data[0] == '%' && data[1] == '%' && data[2] == '%' { if i := p.titleBlockBlock(out, data, true); i > 0 { data = data[i:] continue } } } // blank lines. note: returns the # of bytes to skip if i := p.isEmpty(data); i > 0 { data = data[i:] continue } // indented code block: // // func max(a, b int) int { // if a > b { // return a // } // return b // } if p.codePrefix(data) > 0 { data = data[p.code(out, data):] continue } // fenced code block: // // ``` go // func fact(n int) int { // if n <= 1 { // return n // } // return n * fact(n-1) // } // ``` if p.flags&EXTENSION_FENCED_CODE != 0 { if i := p.fencedCode(out, data, true); i > 0 { data = data[i:] continue } } // horizontal rule: // // ------ // or // ****** // or // ______ if p.isHRule(data) { p.r.HRule(out) var i int for i = 0; data[i] != '\n'; i++ { } data = data[i:] continue } // Aside quote: // // A> This is an aside // A> I found on the web if p.asidePrefix(data) > 0 { data = data[p.aside(out, data):] continue } // Figure "quote": // // F> ![](image) // F> ![](image) // Figure: Caption. if p.figurePrefix(data) > 0 { data = data[p.figure(out, data):] continue } // block quote: // // > A big quote I found somewhere // > on the web if p.quotePrefix(data) > 0 { data = data[p.quote(out, data):] continue } // table: // // Name | Age | Phone // ------|-----|--------- // Bob | 31 | 555-1234 // Alice | 27 | 555-4321 // Table: this is a caption if p.flags&EXTENSION_TABLES != 0 { if i := p.table(out, data); i > 0 { data = data[i:] continue } } // block table: // (cells contain block elements) // // |-------|-----|--------- // | Name | Age | Phone // | ------|-----|--------- // | Bob | 31 | 555-1234 // | Alice | 27 | 555-4321 // |-------|-----|--------- // | Bob | 31 | 555-1234 // | Alice | 27 | 555-4321 if p.flags&EXTENSION_TABLES != 0 { if i := p.blockTable(out, data); i > 0 { data = data[i:] continue } } // a definition list: // // Item1 // : Definition1 // Item2 // : Definition2 if p.dliPrefix(data) > 0 { p.insideDefinitionList = true data = data[p.list(out, data, _LIST_TYPE_DEFINITION, 0, nil):] p.insideDefinitionList = false } // an itemized/unordered list: // // * Item 1 // * Item 2 // // also works with + or - if p.uliPrefix(data) > 0 { data = data[p.list(out, data, 0, 0, nil):] continue } // a numbered/ordered list: // // 1. Item 1 // 2. Item 2 if i := p.oliPrefix(data); i > 0 { start := 0 if i > 2 { start, _ = strconv.Atoi(string(data[:i-2])) // this cannot fail because we just est. the thing *is* a number, and if it does start is zero anyway. } data = data[p.list(out, data, _LIST_TYPE_ORDERED, start, nil):] continue } // a numberd/ordered list: // // ii. Item 1 // ii. Item 2 if p.rliPrefix(data) > 0 { data = data[p.list(out, data, _LIST_TYPE_ORDERED|_LIST_TYPE_ORDERED_ROMAN_LOWER, 0, nil):] continue } // a numberd/ordered list: // // II. Item 1 // II. Item 2 if p.rliPrefixU(data) > 0 { data = data[p.list(out, data, _LIST_TYPE_ORDERED|_LIST_TYPE_ORDERED_ROMAN_UPPER, 0, nil):] continue } // a numberd/ordered list: // // a. Item 1 // b. Item 2 if p.aliPrefix(data) > 0 { data = data[p.list(out, data, _LIST_TYPE_ORDERED|_LIST_TYPE_ORDERED_ALPHA_LOWER, 0, nil):] continue } // a numberd/ordered list: // // A. Item 1 // B. Item 2 if p.aliPrefixU(data) > 0 { data = data[p.list(out, data, _LIST_TYPE_ORDERED|_LIST_TYPE_ORDERED_ALPHA_UPPER, 0, nil):] continue } // an example lists: // // (@good) Item1 // (@good) Item2 if i := p.eliPrefix(data); i > 0 { group := data[2 : i-2] data = data[p.list(out, data, _LIST_TYPE_ORDERED|_LIST_TYPE_ORDERED_GROUP, 0, group):] continue } // anything else must look like a normal paragraph // note: this finds underlined headers, too data = data[p.paragraph(out, data):] } p.nesting-- } func (p *parser) isPrefixHeader(data []byte) bool { // CommonMark: up to three spaces allowed k := 0 for k < len(data) && data[k] == ' ' { k++ } if k == len(data) || k > 3 { return false } data = data[k:] if data[0] != '#' { return false } if p.flags&EXTENSION_SPACE_HEADERS != 0 { level := 0 for level < 6 && data[level] == '#' { level++ } if !iswhitespace(data[level]) { return false } } return true } func (p *parser) prefixHeader(out *bytes.Buffer, data []byte) int { // CommonMark: up to three spaces allowed k := 0 for k < len(data) && data[k] == ' ' { k++ } if k == len(data) || k > 3 { return 0 } data = data[k:] level := 0 for level < 6 && data[level] == '#' { level++ } i, end := 0, 0 for i = level; iswhitespace(data[i]); i++ { } for end = i; data[end] != '\n'; end++ { } skip := end id := "" if p.flags&EXTENSION_HEADER_IDS != 0 { j, k := 0, 0 // find start/end of header id for j = i; j < end-1 && (data[j] != '{' || data[j+1] != '#'); j++ { } for k = j + 1; k < end && data[k] != '}'; k++ { } // extract header id iff found if j < end && k < end { id = string(data[j+2 : k]) end = j skip = k + 1 for end > 0 && data[end-1] == ' ' { end-- } } } // CommonMark spaces *after* the header for end > 0 && data[end-1] == ' ' { end-- } for end > 0 && data[end-1] == '#' { // CommonMark: a # directly following the header name is allowed and we // should keep it if end > 1 && data[end-2] != '#' && !iswhitespace(data[end-2]) { end++ break } end-- } for end > 0 && iswhitespace(data[end-1]) { end-- } if end > i { if id == "" && p.flags&EXTENSION_AUTO_HEADER_IDS != 0 { id = createSanitizedAnchorName(string(data[i:end])) } work := func() bool { p.inline(out, data[i:end]) return true } if id != "" { if v, ok := p.anchors[id]; ok && p.flags&EXTENSION_UNIQUE_HEADER_IDS != 0 { p.anchors[id]++ // anchor found id += "-" + strconv.Itoa(v) } else { p.anchors[id] = 1 } } p.r.SetInlineAttr(p.ial) p.ial = nil p.r.Header(out, work, level, id) } return skip + k } func (p *parser) isUnderlinedHeader(data []byte) int { // test of level 1 header if data[0] == '=' { i := 1 for data[i] == '=' { i++ } for iswhitespace(data[i]) { i++ } if data[i] == '\n' { return 1 } return 0 } // test of level 2 header if data[0] == '-' { i := 1 for data[i] == '-' { i++ } for iswhitespace(data[i]) { i++ } if data[i] == '\n' { return 2 } return 0 } return 0 } func (p *parser) isPartHeader(data []byte) bool { k := 0 for k < len(data) && data[k] == ' ' { k++ } if k == len(data) || k > 3 { return false } data = data[k:] if len(data) < 3 { return false } if data[0] != '-' || data[1] != '#' { return false } if p.flags&EXTENSION_SPACE_HEADERS != 0 { if !iswhitespace(data[2]) { return false } } return true } func (p *parser) isSpecialHeader(data []byte) bool { k := 0 for k < len(data) && data[k] == ' ' { k++ } if k == len(data) || k > 3 { return false } data = data[k:] if len(data) < 3 { return false } if data[0] != '.' || data[1] != '#' { return false } if p.flags&EXTENSION_SPACE_HEADERS != 0 { if !iswhitespace(data[2]) { return false } } return true } func (p *parser) specialHeader(out *bytes.Buffer, data []byte) int { k := 0 for k < len(data) && data[k] == ' ' { k++ } if k == len(data) || k > 3 { return 0 } data = data[k:] if len(data) < 3 { return 0 } if data[0] != '.' || data[1] != '#' { return 0 } i, end := 0, 0 for i = 2; iswhitespace(data[i]); i++ { } for end = i; data[end] != '\n'; end++ { } skip := end id := "" if p.flags&EXTENSION_HEADER_IDS != 0 { j, k := 0, 0 // find start/end of header id for j = i; j < end-1 && (data[j] != '{' || data[j+1] != '#'); j++ { } for k = j + 1; k < end && data[k] != '}'; k++ { } // extract header id iff found if j < end && k < end { id = string(data[j+2 : k]) end = j skip = k + 1 for end > 0 && data[end-1] == ' ' { end-- } } } // CommonMark spaces *after* the header for end > 0 && data[end-1] == ' ' { end-- } // Remove this, not true for this header for end > 0 && data[end-1] == '#' { // CommonMark: a # directly following the header name is allowed and we // should keep it if end > 1 && data[end-2] != '#' && !iswhitespace(data[end-2]) { end++ break } end-- } for end > 0 && iswhitespace(data[end-1]) { end-- } if end > i { if id == "" && p.flags&EXTENSION_AUTO_HEADER_IDS != 0 { id = createSanitizedAnchorName(string(data[i:end])) } work := func() bool { p.inline(out, data[i:end]) return true } p.r.SetInlineAttr(p.ial) p.ial = nil name := bytes.ToLower(data[i:end]) switch { case bytes.Compare(name, []byte("abstract")) == 0: fallthrough case bytes.Compare(name, []byte("preface")) == 0: name := bytes.ToLower(data[i:end]) if id != "" { if v, ok := p.anchors[id]; ok && p.flags&EXTENSION_UNIQUE_HEADER_IDS != 0 { p.anchors[id]++ // anchor found id += "-" + strconv.Itoa(v) } else { p.anchors[id] = 1 } } p.r.SpecialHeader(out, name, work, id) default: // A note section // There is no id for notes, but we still give it to the method. p.r.Note(out, work, id) } } return skip + k } func (p *parser) partHeader(out *bytes.Buffer, data []byte) int { k := 0 for k < len(data) && data[k] == ' ' { k++ } if k == len(data) || k > 3 { return 0 } data = data[k:] if len(data) < 3 { return 0 } if data[0] != '-' || data[1] != '#' { return 0 } i, end := 0, 0 for i = 2; iswhitespace(data[i]); i++ { } for end = i; data[end] != '\n'; end++ { } skip := end id := "" if p.flags&EXTENSION_HEADER_IDS != 0 { j, k := 0, 0 // find start/end of header id for j = i; j < end-1 && (data[j] != '{' || data[j+1] != '#'); j++ { } for k = j + 1; k < end && data[k] != '}'; k++ { } // extract header id iff found if j < end && k < end { id = string(data[j+2 : k]) end = j skip = k + 1 for end > 0 && data[end-1] == ' ' { end-- } } } // CommonMark spaces *after* the header for end > 0 && data[end-1] == ' ' { end-- } for end > 0 && data[end-1] == '#' { // CommonMark: a # directly following the header name is allowed and we // should keep it if end > 1 && data[end-2] != '#' && !iswhitespace(data[end-2]) { end++ break } end-- } for end > 0 && iswhitespace(data[end-1]) { end-- } if end > i { if id == "" && p.flags&EXTENSION_AUTO_HEADER_IDS != 0 { id = createSanitizedAnchorName(string(data[i:end])) } work := func() bool { p.inline(out, data[i:end]) return true } if id != "" { if v, ok := p.anchors[id]; ok && p.flags&EXTENSION_UNIQUE_HEADER_IDS != 0 { p.anchors[id]++ // anchor found id += "-" + strconv.Itoa(v) } else { p.anchors[id] = 1 } } p.r.SetInlineAttr(p.ial) p.ial = nil p.r.Part(out, work, id) } return skip + k } func (p *parser) titleBlock(out *bytes.Buffer, data []byte, doRender bool) int { if p.titleblock { return 0 } if data[0] != '%' { return 0 } splitData := bytes.Split(data, []byte("\n")) var i int for idx, b := range splitData { if !bytes.HasPrefix(b, []byte("%")) { i = idx // - 1 break } } p.titleblock = true data = bytes.Join(splitData[0:i], []byte("\n")) block := p.titleBlockTOML(out, data) p.r.TitleBlockTOML(out, &block) return len(data) } func (p *parser) titleBlockBlock(out *bytes.Buffer, data []byte, doRender bool) int { if p.titleblock { return 0 } if data[0] != '%' || data[1] != '%' || data[2] != '%' { return 0 } // find current eol i := 0 for i < len(data) && data[i] != '\n' { i++ } beg := i delimLength := 0 for i < len(data) { if delimLength = p.isTOMLBlockBlock(data[i:]); delimLength > 0 { break } i++ } end := i p.titleblock = true data = data[beg:end] block := p.titleBlockTOML(out, data) p.r.TitleBlockTOML(out, &block) return len(data) + delimLength + beg } func (p *parser) html(out *bytes.Buffer, data []byte, doRender bool) int { var i, j int // identify the opening tag if data[0] != '<' { return 0 } curtag, tagfound := p.htmlFindTag(data[1:]) // handle special cases if !tagfound { // check for an HTML comment if size := p.htmlComment(out, data, doRender); size > 0 { return size } // check for an
tag if size := p.htmlHr(out, data, doRender); size > 0 { return size } // check for HTML CDATA if size := p.htmlCDATA(out, data, doRender); size > 0 { return size } // check for an if size := p.htmlReference(out, data, doRender); size > 0 { return size } // no special case recognized return 0 } // look for an unindented matching closing tag // followed by a blank line found := false // if not found, try a second pass looking for indented match // but not if tag is "ins" or "del" (following original Markdown.pl) if !found && curtag != "ins" && curtag != "del" { i = 1 for i < len(data) { i++ for i < len(data) && !(data[i-1] == '<' && data[i] == '/') { i++ } if i+2+len(curtag) >= len(data) { break } j = p.htmlFindEnd(curtag, data[i-1:]) if j > 0 { i += j - 1 found = true break } } } if !found { return 0 } // the end of the block has been found if doRender { // trim newlines end := i for end > 0 && data[end-1] == '\n' { end-- } p.r.BlockHtml(out, data[:end]) } return i } func (p *parser) renderHTMLBlock(out *bytes.Buffer, data []byte, start int, doRender bool) int { // html block needs to end with a blank line if i := p.isEmpty(data[start:]); i > 0 { size := start + i if doRender { // trim trailing newlines end := size for end > 0 && data[end-1] == '\n' { end-- } // breaks the tests if we parse this // var cooked bytes.Buffer // p.inline(&cooked, data[:end]) p.r.SetInlineAttr(p.ial) p.ial = nil p.r.CommentHtml(out, data[:end]) } return size } return 0 } // HTML comment, lax form func (p *parser) htmlComment(out *bytes.Buffer, data []byte, doRender bool) int { i := p.inlineHTMLComment(out, data) return p.renderHTMLBlock(out, data, i, doRender) } // HTML CDATA section func (p *parser) htmlCDATA(out *bytes.Buffer, data []byte, doRender bool) int { const cdataTag = "') { i++ } i++ // no end-of-comment marker if i >= len(data) { return 0 } return p.renderHTMLBlock(out, data, i, doRender) } // HR, which is the only self-closing block tag considered func (p *parser) htmlHr(out *bytes.Buffer, data []byte, doRender bool) int { if data[0] != '<' || (data[1] != 'h' && data[1] != 'H') || (data[2] != 'r' && data[2] != 'R') { return 0 } if data[3] != ' ' && data[3] != '/' && data[3] != '>' { // not an
tag after all; at least not a valid one return 0 } i := 3 for data[i] != '>' && data[i] != '\n' { i++ } if data[i] == '>' { return p.renderHTMLBlock(out, data, i+1, doRender) } return 0 } // HTML reference, actually xml, but keep the spirit and call it html func (p *parser) htmlReference(out *bytes.Buffer, data []byte, doRender bool) int { if !bytes.HasPrefix(data, []byte("') { i++ } i++ // no end-of-reference marker if i >= len(data) { return 0 } // needs to end with a blank line if j := p.isEmpty(data[i:]); j > 0 { size := i + j if doRender { // trim trailing newlines end := size for end > 0 && data[end-1] == '\n' { end-- } anchor := bytes.Index(data[:end], []byte("anchor=")) if anchor == -1 { // nothing found, not a real reference return 0 } // look for the some tag after anchor= open := data[anchor+7] i := anchor + 7 + 2 for i < end && data[i-1] != open { i++ } if i >= end { return 0 } anchorStr := string(data[anchor+7+1 : i-1]) if c, ok := p.citations[anchorStr]; !ok { p.citations[anchorStr] = &citation{xml: data[:end]} } else { c.xml = data[:end] } } return size } return 0 } func (p *parser) htmlFindTag(data []byte) (string, bool) { i := 0 for isalnum(data[i]) { i++ } key := string(data[:i]) if _, ok := blockTags[key]; ok { return key, true } return "", false } func (p *parser) htmlFindEnd(tag string, data []byte) int { // assume data[0] == '<' && data[1] == '/' already tested // check if tag is a match closetag := []byte("") if !bytes.HasPrefix(data, closetag) { return 0 } i := len(closetag) // check that the rest of the line is blank skip := 0 if skip = p.isEmpty(data[i:]); skip == 0 { return 0 } i += skip skip = 0 if i >= len(data) { return i } if p.flags&EXTENSION_LAX_HTML_BLOCKS != 0 { return i } if skip = p.isEmpty(data[i:]); skip == 0 { // following line must be blank return 0 } return i + skip } func (p *parser) isEmpty(data []byte) int { // it is okay to call isEmpty on an empty buffer if len(data) == 0 { return 0 } var i int for i = 0; i < len(data) && data[i] != '\n' && data[i] != '\r'; i++ { if data[i] != ' ' && data[i] != '\t' { return 0 } } return i + 1 } func (p *parser) isHRule(data []byte) bool { i := 0 // skip up to three spaces for i < 3 && data[i] == ' ' { i++ } // look at the hrule char if data[i] != '*' && data[i] != '-' && data[i] != '_' { return false } c := data[i] // the whole line must be the char or whitespace n := 0 for data[i] != '\n' { switch { case data[i] == c: n++ case data[i] != ' ': return false } i++ } return n >= 3 } func (p *parser) isFencedCode(data []byte, syntax **string, oldmarker string) (skip int, marker string) { i, size := 0, 0 skip = 0 // skip up to three spaces for i < len(data) && i < 3 && data[i] == ' ' { i++ } if i >= len(data) { return } // check for the marker characters: ~ or ` if data[i] != '~' && data[i] != '`' { return } c := data[i] // the whole line must be the same char or whitespace for i < len(data) && data[i] == c { size++ i++ } if i >= len(data) { return } // the marker char must occur at least 3 times if size < 3 { return } marker = string(data[i-size : i]) // if this is the end marker, it must match the beginning marker if oldmarker != "" && marker != oldmarker { return } if syntax != nil { syn := 0 i = skipChar(data, i, ' ') if i >= len(data) { return } syntaxStart := i if data[i] == '{' { i++ syntaxStart++ for i < len(data) && data[i] != '}' && data[i] != '\n' { syn++ i++ } if i >= len(data) && data[i] != '}' { return } // strip all whitespace at the beginning and the end // of the {} block for syn > 0 && isspace(data[syntaxStart]) { syntaxStart++ syn-- } for syn > 0 && isspace(data[syntaxStart+syn-1]) { syn-- } i++ } else { for i < len(data) && !isspace(data[i]) { syn++ i++ } } language := string(data[syntaxStart : syntaxStart+syn]) *syntax = &language } i = skipChar(data, i, ' ') if i >= len(data) || data[i] != '\n' { return } skip = i + 1 return } func (p *parser) isTOMLBlockBlock(data []byte) int { if len(data) < 3 { return 0 } if data[0] != '%' || data[1] != '%' || data[2] != '%' { return 0 } i := 0 for data[i] != '\n' { switch { case data[i] == '%': i++ case data[i] != ' ': return 0 } i++ } return i } func (p *parser) fencedCode(out *bytes.Buffer, data []byte, doRender bool) int { var lang *string beg, marker := p.isFencedCode(data, &lang, "") if beg == 0 || beg >= len(data) { return 0 } co := "" if p.ial != nil { // enabled, any non-empty value co = p.ial.Value("callout") } // CommonMark: if indented strip this many leading spaces from code block indent := 0 for indent < beg && data[indent] == ' ' { indent++ } var work bytes.Buffer for { // safe to assume beg < len(data) // check for the end of the code block fenceEnd, _ := p.isFencedCode(data[beg:], nil, marker) if fenceEnd != 0 { beg += fenceEnd break } // copy the current line end := beg for end < len(data) && data[end] != '\n' { end++ } end++ // did we reach the end of the buffer without a closing marker? if end >= len(data) { return 0 } // CommmonMark, strip beginning spaces s := 0 for s < indent && data[beg] == ' ' { beg++ s++ } // verbatim copy to the working buffer if doRender { work.Write(data[beg:end]) } beg = end } var caption bytes.Buffer line := beg j := beg if bytes.HasPrefix(bytes.TrimSpace(data[j:]), []byte("Figure: ")) { for line < len(data) { j++ // find the end of this line for data[j-1] != '\n' { j++ } if p.isEmpty(data[line:j]) > 0 { break } line = j } if beg+8 < j-1 { p.inline(&caption, data[beg+8:j-1]) } } syntax := "" if lang != nil { syntax = *lang } if doRender { p.r.SetInlineAttr(p.ial) p.ial = nil if co != "" { var callout bytes.Buffer callouts(p, &callout, work.Bytes(), 0, co) p.r.BlockCode(out, callout.Bytes(), syntax, caption.Bytes(), p.insideFigure, true) } else { p.callouts = nil p.r.BlockCode(out, work.Bytes(), syntax, caption.Bytes(), p.insideFigure, false) } } return j } func (p *parser) table(out *bytes.Buffer, data []byte) int { var ( header bytes.Buffer body bytes.Buffer footer bytes.Buffer ) i, columns := p.tableHeader(&header, data) if i == 0 { return 0 } foot := false for i < len(data) { if j := p.isTableFooter(data[i:]); j > 0 && !foot { foot = true i += j continue } pipes, rowStart := 0, i for ; data[i] != '\n'; i++ { if data[i] == '|' { pipes++ } } if pipes == 0 { i = rowStart break } // include the newline in data sent to tableRow i++ if foot { p.tableRow(&footer, data[rowStart:i], columns, false) continue } p.tableRow(&body, data[rowStart:i], columns, false) } var caption bytes.Buffer line := i j := i if bytes.HasPrefix(data[j:], []byte("Table: ")) { for line < len(data) { j++ // find the end of this line for data[j-1] != '\n' { j++ } if p.isEmpty(data[line:j]) > 0 { break } line = j } if i+7 < j-1 { p.inline(&caption, data[i+7:j-1]) // +7 for 'Table: ' } } p.r.SetInlineAttr(p.ial) p.ial = nil p.r.Table(out, header.Bytes(), body.Bytes(), footer.Bytes(), columns, caption.Bytes()) return j } func (p *parser) blockTable(out *bytes.Buffer, data []byte) int { var ( header bytes.Buffer body bytes.Buffer footer bytes.Buffer rowWork bytes.Buffer ) i := p.isBlockTableHeader(data) if i == 0 || i == len(data) { return 0 } j, columns := p.tableHeader(&header, data[i:]) i += j // each cell in a row gets multiple lines which we store per column, we // process the buffers when we see a row separator bodies := make([]bytes.Buffer, len(columns)) colspans := make([]int, len(columns)) foot := false j = 0 for i < len(data) { if j = p.isTableFooter(data[i:]); j > 0 && !foot { // prepare previous ones foot = true i += j continue } if j = p.isRowSeperator(data[i:]); j > 0 { switch foot { case false: // separator before any footer var cellWork bytes.Buffer colSpanSkip := 0 for c := 0; c < len(columns); c++ { cellWork.Truncate(0) if bodies[c].Len() > 0 { p.block(&cellWork, bodies[c].Bytes()) bodies[c].Truncate(0) } if colSpanSkip == 0 { p.r.TableCell(&rowWork, cellWork.Bytes(), columns[c], colspans[c]) } if colspans[c] > 1 { colSpanSkip += colspans[c] } if colSpanSkip > 0 { colSpanSkip-- } } p.r.TableRow(&body, rowWork.Bytes()) rowWork.Truncate(0) i += j continue case true: // closing separator that closes the table i += j continue } } pipes, rowStart := 0, i for ; data[i] != '\n'; i++ { if data[i] == '|' { pipes++ } } if pipes == 0 { i = rowStart break } // include the newline in data sent to tableRow and blockTabeRow i++ if foot { // footer can't contain block level elements p.tableRow(&footer, data[rowStart:i], columns, false) } else { p.blockTableRow(bodies, colspans, data[rowStart:i]) } } // are there cells left to process? if len(bodies) > 0 && bodies[0].Len() != 0 { colSpanSkip := 0 for c := 0; c < len(columns); c++ { var cellWork bytes.Buffer cellWork.Truncate(0) if bodies[c].Len() > 0 { p.block(&cellWork, bodies[c].Bytes()) bodies[c].Truncate(0) } if colSpanSkip == 0 { p.r.TableCell(&rowWork, cellWork.Bytes(), columns[c], colspans[c]) } if colspans[c] > 1 { colSpanSkip += colspans[c] } if colSpanSkip > 0 { colSpanSkip-- } } p.r.TableRow(&body, rowWork.Bytes()) } var caption bytes.Buffer line := i j = i if bytes.HasPrefix(data[j:], []byte("Table: ")) { for line < len(data) { j++ // find the end of this line for data[j-1] != '\n' { j++ } if p.isEmpty(data[line:j]) > 0 { break } line = j } p.inline(&caption, data[i+7:j-1]) // +7 for 'Table: ' } p.r.SetInlineAttr(p.ial) p.ial = nil p.r.Table(out, header.Bytes(), body.Bytes(), footer.Bytes(), columns, caption.Bytes()) return j } // check if the specified position is preceeded by an odd number of backslashes func isBackslashEscaped(data []byte, i int) bool { backslashes := 0 for i-backslashes-1 >= 0 && data[i-backslashes-1] == '\\' { backslashes++ } return backslashes&1 == 1 } func (p *parser) tableHeader(out *bytes.Buffer, data []byte) (size int, columns []int) { i := 0 colCount := 1 for i = 0; data[i] != '\n'; i++ { if data[i] == '|' && !isBackslashEscaped(data, i) { colCount++ } } // doesn't look like a table header if colCount == 1 { return } // include the newline in the data sent to tableRow header := data[:i+1] // column count ignores pipes at beginning or end of line if data[0] == '|' { colCount-- } if i > 2 && data[i-1] == '|' && !isBackslashEscaped(data, i-1) { colCount-- } columns = make([]int, colCount) // move on to the header underline i++ if i >= len(data) { return } if data[i] == '|' && !isBackslashEscaped(data, i) { i++ } for data[i] == ' ' { i++ } // each column header is of form: / *:?-+:? *|/ with # dashes + # colons >= 3 // and trailing | optional on last column col := 0 for data[i] != '\n' { dashes := 0 if data[i] == ':' { i++ columns[col] |= _TABLE_ALIGNMENT_LEFT dashes++ } for data[i] == '-' { i++ dashes++ } if data[i] == ':' { i++ columns[col] |= _TABLE_ALIGNMENT_RIGHT dashes++ } for data[i] == ' ' { i++ } // end of column test is messy switch { case dashes < 3: // not a valid column return case data[i] == '|' && !isBackslashEscaped(data, i): // marker found, now skip past trailing whitespace col++ i++ for data[i] == ' ' { i++ } // trailing junk found after last column if col >= colCount && data[i] != '\n' { return } case (data[i] != '|' || isBackslashEscaped(data, i)) && col+1 < colCount: // something else found where marker was required return case data[i] == '\n': // marker is optional for the last column col++ default: // trailing junk found after last column return } } if col != colCount { return } p.tableRow(out, header, columns, true) size = i + 1 return } func (p *parser) tableRow(out *bytes.Buffer, data []byte, columns []int, header bool) { i, col := 0, 0 var rowWork bytes.Buffer if data[i] == '|' && !isBackslashEscaped(data, i) { i++ } colSpanSkip := 0 for col = 0; col < len(columns) && i < len(data); col++ { for data[i] == ' ' { i++ } cellStart := i for (data[i] != '|' || isBackslashEscaped(data, i)) && data[i] != '\n' { i++ } cellEnd := i // count number of pipe symbols to calculate colspan colspan := 0 for data[i+colspan] == '|' && i+colspan < len(data) { colspan++ } // skip the end-of-cell marker, possibly taking us past end of buffer i++ for cellEnd > cellStart && data[cellEnd-1] == ' ' { cellEnd-- } var cellWork bytes.Buffer p.inline(&cellWork, data[cellStart:cellEnd]) if header { if colSpanSkip == 0 { p.r.TableHeaderCell(&rowWork, cellWork.Bytes(), columns[col], colspan) } } else { if colSpanSkip == 0 { p.r.TableCell(&rowWork, cellWork.Bytes(), columns[col], colspan) } } if colspan > 1 { colSpanSkip += colspan } if colSpanSkip > 0 { colSpanSkip-- } } // pad it out with empty columns to get the right number for ; col < len(columns); col++ { if header { p.r.TableHeaderCell(&rowWork, nil, columns[col], 0) } else { p.r.TableCell(&rowWork, nil, columns[col], 0) } } // silently ignore rows with too many cells p.r.TableRow(out, rowWork.Bytes()) } func (p *parser) blockTableRow(out []bytes.Buffer, colspans []int, data []byte) { i, col := 0, 0 if data[i] == '|' && !isBackslashEscaped(data, i) { i++ } for col = 0; col < len(out) && i < len(data); col++ { for data[i] == ' ' { i++ } cellStart := i for (data[i] != '|' || isBackslashEscaped(data, i)) && data[i] != '\n' { i++ } cellEnd := i // count number of pipe symbols to calculate colspan colspan := 0 for data[i+colspan] == '|' && i+colspan < len(data) { colspan++ } // skip the end-of-cell marker, possibly taking us past end of buffer i++ for cellEnd > cellStart && data[cellEnd-1] == ' ' { cellEnd-- } out[col].Write(data[cellStart:cellEnd]) out[col].WriteByte('\n') colspans[col] = colspan } } // optional | or + at the beginning, then at least 3 equals func (p *parser) isTableFooter(data []byte) int { i := 0 if data[i] == '|' || data[i] == '+' { i++ } if len(data[i:]) < 4 { return 0 } if data[i+1] != '=' && data[i+2] != '=' && data[i+3] != '=' { return 0 } for i < len(data) && data[i] != '\n' { i++ } return i + 1 } // this starts a table, basically three dashes with mandatory | or + at the start func (p *parser) isBlockTableHeader(data []byte) int { i := 0 if data[i] != '|' && data[i] != '+' { return 0 } i++ if len(data[i:]) < 4 { return 0 } if data[i+1] != '-' && data[i+2] != '-' && data[i+3] != '-' { return 0 } for i < len(data) && data[i] != '\n' { i++ } return i + 1 } // table row separator (use in block tables): | or + at the start, 3 or more dashes func (p *parser) isRowSeperator(data []byte) int { return p.isBlockTableHeader(data) } // returns prefix length for block code func (p *parser) codePrefix(data []byte) int { if data[0] == ' ' && data[1] == ' ' && data[2] == ' ' && data[3] == ' ' { return 4 } return 0 } func (p *parser) code(out *bytes.Buffer, data []byte) int { var work bytes.Buffer i := 0 for i < len(data) { beg := i for data[i] != '\n' { i++ } i++ blankline := p.isEmpty(data[beg:i]) > 0 if pre := p.codePrefix(data[beg:i]); pre > 0 { beg += pre } else if !blankline { // non-empty, non-prefixed line breaks the pre i = beg break } // verbatim copy to the working buffer if blankline { work.WriteByte('\n') } else { work.Write(data[beg:i]) } } caption := "" line := i j := i // In the case of F> there may be spaces in front of it if bytes.HasPrefix(bytes.TrimSpace(data[j:]), []byte("Figure: ")) { for line < len(data) { j++ // find the end of this line for data[j-1] != '\n' { j++ } if p.isEmpty(data[line:j]) > 0 { break } line = j } // save for later processing. if i+8 < j-1 { caption = string(data[i+8 : j-1]) // +8 for 'Figure: ' } } // trim all the \n off the end of work workbytes := work.Bytes() eol := len(workbytes) for eol > 0 && workbytes[eol-1] == '\n' { eol-- } if eol != len(workbytes) { work.Truncate(eol) } work.WriteByte('\n') co := "" if p.ial != nil { // enabled, any non-empty value co = p.ial.Value("callout") } p.r.SetInlineAttr(p.ial) p.ial = nil var capb bytes.Buffer if co != "" { var callout bytes.Buffer callouts(p, &callout, work.Bytes(), 0, co) p.inline(&capb, []byte(caption)) p.r.BlockCode(out, callout.Bytes(), "", capb.Bytes(), p.insideFigure, true) } else { p.callouts = nil p.inline(&capb, []byte(caption)) p.r.BlockCode(out, work.Bytes(), "", capb.Bytes(), p.insideFigure, false) } return j } // returns unordered list item prefix func (p *parser) uliPrefix(data []byte) int { i := 0 if len(data) < 3 { return 0 } // start with up to 3 spaces for i < 3 && data[i] == ' ' { i++ } // need a *, +, #, or - followed by a space if (data[i] != '*' && data[i] != '+' && data[i] != '-' && !iswhitespace(data[i])) || !iswhitespace(data[i+1]) { return 0 } return i + 2 } // returns ordered list item prefix func (p *parser) oliPrefix(data []byte) int { i := 0 if len(data) < 3 { return 0 } // start with up to 3 spaces for i < 3 && data[i] == ' ' { i++ } // count the digits start := i for isnum(data[i]) { i++ } // we need >= 1 digits followed by a dot or brace and a space if start == i || (data[i] != '.' && data[i] != ')') || !iswhitespace(data[i+1]) { return 0 } return i + 2 } // returns ordered list item prefix for alpha ordered list func (p *parser) aliPrefix(data []byte) int { i := 0 if len(data) < 4 { return 0 } // start with up to 3 spaces for i < 3 && data[i] == ' ' { i++ } // count the digits start := i for data[i] >= 'a' && data[i] <= 'z' { i++ } // we need >= 1 letter followed by a dot and two spaces if start == i || (data[i] != '.' && data[i] != ')') || !iswhitespace(data[i+1]) || !iswhitespace(data[i+2]) { return 0 } if i-start > 2 { // crazy list, i.e. too many letters. return 0 } return i + 3 } // returns ordered list item prefix for alpha uppercase ordered list func (p *parser) aliPrefixU(data []byte) int { i := 0 if len(data) < 4 { return 0 } // start with up to 3 spaces for i < 3 && data[i] == ' ' { i++ } // count the digits start := i for isupper(data[i]) { i++ } // we need >= 1 letter followed by a dot and two spaces if start == i || (data[i] != '.' && data[i] != ')') || !iswhitespace(data[i+1]) || !iswhitespace(data[i+2]) { return 0 } if i-start > 2 { // crazy list, i.e. too many letters. return 0 } return i + 3 } // returns ordered list item prefix for roman ordered list func (p *parser) rliPrefix(data []byte) int { i := 0 if len(data) < 4 { return 0 } // start with up to 3 spaces for i < 3 && data[i] == ' ' { i++ } // count the digits start := i for isroman(data[i], false) { i++ } // we need >= 1 letter followed by a dot and two spaces if start == i || (data[i] != '.' && data[i] != ')') || !iswhitespace(data[i+1]) || !iswhitespace(data[i+2]) { return 0 } return i + 3 } // returns ordered list item prefix for roman uppercase ordered list func (p *parser) rliPrefixU(data []byte) int { i := 0 if len(data) < 4 { return 0 } // start with up to 3 spaces for i < 3 && data[i] == ' ' { i++ } // count the digits start := i for isroman(data[i], true) { i++ } // we need >= 1 letter followed by a dot and two spaces if start == i || (data[i] != '.' && data[i] != ')') || !iswhitespace(data[i+1]) || !iswhitespace(data[i+2]) { return 0 } return i + 3 } // returns definition list item prefix func (p *parser) dliPrefix(data []byte) int { // return the index of where the term ends i := 0 for data[i] != '\n' && i < len(data) { i++ } if i == 0 || i == len(data) { return 0 } // allow an optional newline to be here if i+1 < len(data) && data[i] == '\n' && data[i+1] == '\n' { i++ } // start with up to 3 spaces before : j := 0 for j < 3 && iswhitespace(data[i+j]) && i+j < len(data) { j++ } i += j + 1 if i >= len(data) { return 0 } if data[i] == ':' { return i + 1 } return 0 } // returns example list item prefix func (p *parser) eliPrefix(data []byte) int { i := 0 if len(data) < 6 { return 0 } // start with up to 3 spaces for i < 3 && iswhitespace(data[i]) { i++ } // (@) if data[i] != '(' || data[i+1] != '@' { return 0 } // count up until the closing ) for data[i] != ')' { i++ if i == len(data) { return 0 } } // now two spaces if data[i] != ')' || !iswhitespace(data[i+1]) || !iswhitespace(data[i+2]) { return 0 } return i + 2 } // parse ordered or unordered or definition list block func (p *parser) list(out *bytes.Buffer, data []byte, flags, start int, group []byte) int { p.insideList++ defer func() { p.insideList-- }() i := 0 flags |= _LIST_ITEM_BEGINNING_OF_LIST work := func() bool { for i < len(data) { skip := p.listItem(out, data[i:], &flags) i += skip if skip == 0 || flags&_LIST_ITEM_END_OF_LIST != 0 { break } flags &= ^_LIST_ITEM_BEGINNING_OF_LIST } return true } if group != nil { gr := string(group) if _, ok := p.examples[gr]; ok { p.examples[gr]++ } else { p.examples[gr] = 1 } } p.r.SetInlineAttr(p.ial) p.ial = nil if p.insideList > 1 { flags |= _LIST_INSIDE_LIST } else { flags &= ^_LIST_INSIDE_LIST } p.r.List(out, work, flags, start, group) return i } // Parse a single list item. // Assumes initial prefix is already removed if this is a sublist. func (p *parser) listItem(out *bytes.Buffer, data []byte, flags *int) int { // keep track of the indentation of the first line itemIndent := 0 for itemIndent < 3 && iswhitespace(data[itemIndent]) { itemIndent++ } i := p.uliPrefix(data) if i == 0 { i = p.oliPrefix(data) } if i == 0 { i = p.aliPrefix(data) } if i == 0 { i = p.aliPrefixU(data) } if i == 0 { i = p.rliPrefix(data) } if i == 0 { i = p.rliPrefixU(data) } if i == 0 { i = p.eliPrefix(data) } if i == 0 { i = p.dliPrefix(data) if i > 0 { var rawTerm bytes.Buffer if data[i-2] == '\n' && data[i-3] == '\n' { p.inline(&rawTerm, data[:i-3]) // -3 for : and the 2newline } else { p.inline(&rawTerm, data[:i-2]) // -2 for : and the newline } p.r.ListItem(out, rawTerm.Bytes(), *flags|_LIST_TYPE_TERM) } } if i == 0 { return 0 } // skip leading whitespace on first line for iswhitespace(data[i]) { i++ } // find the end of the line line := i for data[i-1] != '\n' { i++ } // get working buffer var raw bytes.Buffer // put the first line into the working buffer raw.Write(data[line:i]) line = i // process the following lines containsBlankLine := false sublist := 0 gatherlines: for line < len(data) { i++ // find the end of this line for data[i-1] != '\n' { i++ } // if it is an empty line, guess that it is part of this item // and move on to the next line if p.isEmpty(data[line:i]) > 0 { containsBlankLine = true raw.Write(data[line:i]) line = i continue } // calculate the indentation indent := 0 for indent < 4 && line+indent < i && data[line+indent] == ' ' { indent++ } chunk := data[line+indent : i] // evaluate how this line fits in switch { // is this a nested list item? case (p.uliPrefix(chunk) > 0 && !p.isHRule(chunk)) || p.aliPrefix(chunk) > 0 || p.aliPrefixU(chunk) > 0 || p.rliPrefix(chunk) > 0 || p.rliPrefixU(chunk) > 0 || p.oliPrefix(chunk) > 0 || p.eliPrefix(chunk) > 0 || p.dliPrefix(data[line+indent:]) > 0: if containsBlankLine { *flags |= _LIST_ITEM_CONTAINS_BLOCK } // to be a nested list, it must be indented more // if not, it is the next item in the same list if indent <= itemIndent { break gatherlines } // is this the first item in the the nested list? if sublist == 0 { sublist = raw.Len() } // is this a nested prefix header? case p.isPrefixHeader(chunk): // if the header is not indented, it is not nested in the list // and thus ends the list if containsBlankLine && indent < 4 { *flags |= _LIST_ITEM_END_OF_LIST break gatherlines } *flags |= _LIST_ITEM_CONTAINS_BLOCK // anything following an empty line is only part // of this item if it is indented 4 spaces // (regardless of the indentation of the beginning of the item) // if the is beginning with ': term', we have a new term case containsBlankLine && indent < 4: *flags |= _LIST_ITEM_END_OF_LIST break gatherlines // a blank line means this should be parsed as a block case containsBlankLine: *flags |= _LIST_ITEM_CONTAINS_BLOCK // CommonMark, rule breaks the list, but when indented it belong to the list case p.isHRule(chunk) && indent < 4: *flags |= _LIST_ITEM_END_OF_LIST break gatherlines } containsBlankLine = false // add the line into the working buffer without prefix raw.Write(data[line+indent : i]) line = i } rawBytes := raw.Bytes() // render the contents of the list item var cooked bytes.Buffer if *flags&_LIST_ITEM_CONTAINS_BLOCK != 0 { // intermediate render of block li if sublist > 0 { p.block(&cooked, rawBytes[:sublist]) p.block(&cooked, rawBytes[sublist:]) } else { p.block(&cooked, rawBytes) } } else { // intermediate render of inline li if sublist > 0 { p.inline(&cooked, rawBytes[:sublist]) p.block(&cooked, rawBytes[sublist:]) } else { p.inline(&cooked, rawBytes) } } // render the actual list item cookedBytes := cooked.Bytes() parsedEnd := len(cookedBytes) // strip trailing newlines for parsedEnd > 0 && cookedBytes[parsedEnd-1] == '\n' { parsedEnd-- } p.r.ListItem(out, cookedBytes[:parsedEnd], *flags) return line } // render a single paragraph that has already been parsed out func (p *parser) renderParagraph(out *bytes.Buffer, data []byte) { if len(data) == 0 { return } // trim leading spaces beg := 0 for iswhitespace(data[beg]) { beg++ } // trim trailing newline end := len(data) - 1 // trim trailing spaces for end > beg && iswhitespace(data[end-1]) { end-- } if isMatter(data) { p.inline(out, data[beg:end]) return } p.displayMath = false work := func() bool { // if we are a single paragraph constisting entirely out of math // we set the displayMath to true k := 0 if end-beg > 4 && data[beg] == '$' && data[beg+1] == '$' { for k = beg + 2; k < end-1; k++ { if data[k] == '$' && data[k+1] == '$' { break } } if k+2 == end { p.displayMath = true } } p.inline(out, data[beg:end]) return true } flags := 0 if p.insideDefinitionList { flags |= _LIST_TYPE_DEFINITION } if p.insideList > 0 { flags |= _LIST_INSIDE_LIST // Not really, just in a list } else { flags &= ^_LIST_INSIDE_LIST // Not really, just in a list } p.r.Paragraph(out, work, flags) } func (p *parser) paragraph(out *bytes.Buffer, data []byte) int { // prev: index of 1st char of previous line // line: index of 1st char of current line // i: index of cursor/end of current line var prev, line, i int // keep going until we find something to mark the end of the paragraph for i < len(data) { // mark the beginning of the current line prev = line current := data[i:] line = i // did we find a blank line marking the end of the paragraph? if n := p.isEmpty(current); n > 0 { p.renderParagraph(out, data[:i]) return i + n } // an underline under some text marks a header, so our paragraph ended on prev line if i > 0 { if level := p.isUnderlinedHeader(current); level > 0 { // render the paragraph p.renderParagraph(out, data[:prev]) // ignore leading and trailing whitespace eol := i - 1 for prev < eol && iswhitespace(data[prev]) { prev++ } for eol > prev && iswhitespace(data[eol-1]) { eol-- } // render the header // this ugly double closure avoids forcing variables onto the heap work := func(o *bytes.Buffer, pp *parser, d []byte) func() bool { return func() bool { // this renders the name, but how to make attribute out of it pp.inline(o, d) return true } }(out, p, data[prev:eol]) id := "" if p.flags&EXTENSION_AUTO_HEADER_IDS != 0 { id = createSanitizedAnchorName(string(data[prev:eol])) } p.r.SetInlineAttr(p.ial) p.ial = nil p.r.Header(out, work, level, id) // find the end of the underline for data[i] != '\n' { i++ } return i } } // if the next line starts a block of HTML, then the paragraph ends here if p.flags&EXTENSION_LAX_HTML_BLOCKS != 0 { if data[i] == '<' && p.html(out, current, false) > 0 { // rewind to before the HTML block p.renderParagraph(out, data[:i]) return i } } // if there's a prefixed header or a horizontal rule after this, paragraph is over if p.isPrefixHeader(current) || p.isHRule(current) { p.renderParagraph(out, data[:i]) return i } // if there's a fenced code block, paragraph is over if p.flags&EXTENSION_FENCED_CODE != 0 { if p.fencedCode(out, current, false) > 0 { p.renderParagraph(out, data[:i]) return i } } // if there's a list after this, paragraph is over if p.flags&EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK != 0 { if p.uliPrefix(current) != 0 || p.aliPrefixU(current) != 0 || p.aliPrefix(current) != 0 || p.rliPrefixU(current) != 0 || p.rliPrefix(current) != 0 || p.oliPrefix(current) != 0 || p.eliPrefix(current) != 0 || p.dliPrefix(current) != 0 || p.quotePrefix(current) != 0 || p.figurePrefix(current) != 0 || p.asidePrefix(current) != 0 || p.codePrefix(current) != 0 { p.renderParagraph(out, data[:i]) return i } } // otherwise, scan to the beginning of the next line for data[i] != '\n' { i++ } i++ } p.renderParagraph(out, data[:i]) return i } func createSanitizedAnchorName(text string) string { var anchorName []rune number := 0 for _, r := range []rune(text) { switch { case r == ' ': anchorName = append(anchorName, '-') case unicode.IsNumber(r): number++ fallthrough case unicode.IsLetter(r): anchorName = append(anchorName, unicode.ToLower(r)) } } if number == len(anchorName) { anchorName = append([]rune("section-"), anchorName...) } return string(anchorName) } func isMatter(text []byte) bool { if string(text) == "{frontmatter}\n" { return true } if string(text) == "{mainmatter}\n" { return true } if string(text) == "{backmatter}\n" { return true } return false } mmark-1.3-git849458a/block_test.go000066400000000000000000001505551264073172300166140ustar00rootroot00000000000000// Unit tests for block parsing package mmark import ( "io/ioutil" "os" "strings" "testing" ) func init() { test = true } func runMarkdownBlock(input string, extensions int) string { htmlFlags := 0 renderer := HtmlRenderer(htmlFlags, "", "") return Parse([]byte(input), renderer, extensions).String() } func runMarkdownBlockXML(input string, extensions int) string { xmlFlags := 0 extensions |= commonXmlExtensions extensions |= EXTENSION_UNIQUE_HEADER_IDS renderer := XmlRenderer(xmlFlags) return Parse([]byte(input), renderer, extensions).String() } func doTestsBlock(t *testing.T, tests []string, extensions int) { // catch and report panics var candidate string defer func() { if err := recover(); err != nil { t.Errorf("\npanic while processing [%#v]: %s\n", candidate, err) } }() for i := 0; i+1 < len(tests); i += 2 { input := tests[i] candidate = input expected := tests[i+1] actual := runMarkdownBlock(candidate, extensions) if actual != expected { t.Errorf("\nInput [%#v]\nExpected[%#v]\nActual [%#v]", candidate, expected, actual) } // now test every substring to stress test bounds checking if !testing.Short() { for start := 0; start < len(input); start++ { for end := start + 1; end <= len(input); end++ { candidate = input[start:end] _ = runMarkdownBlock(candidate, extensions) } } } } } func doTestsBlockXML(t *testing.T, tests []string, extensions int) { // catch and report panics var candidate string defer func() { if err := recover(); err != nil { t.Errorf("\npanic while processing [%#v]: %s\n", candidate, err) } }() for i := 0; i+1 < len(tests); i += 2 { input := tests[i] candidate = input expected := tests[i+1] actual := runMarkdownBlockXML(candidate, extensions) if actual != expected { t.Errorf("\nInput [%#v]\nExpected[%#v]\nActual [%#v]", candidate, expected, actual) } // now test every substring to stress test bounds checking if !testing.Short() { for start := 0; start < len(input); start++ { for end := start + 1; end <= len(input); end++ { candidate = input[start:end] _ = runMarkdownBlockXML(candidate, extensions) } } } } } func TestPrefixHeaderNoExtensions(t *testing.T) { var tests = []string{ "# Header 1\n", "

Header 1

\n", "## Header 2\n", "

Header 2

\n", "### Header 3\n", "

Header 3

\n", "#### Header 4\n", "

Header 4

\n", "##### Header 5\n", "
Header 5
\n", "###### Header 6\n", "
Header 6
\n", "####### Header 7\n", "
# Header 7
\n", "#Header 1\n", "

Header 1

\n", "##Header 2\n", "

Header 2

\n", "###Header 3\n", "

Header 3

\n", "####Header 4\n", "

Header 4

\n", "#####Header 5\n", "
Header 5
\n", "######Header 6\n", "
Header 6
\n", "#######Header 7\n", "
#Header 7
\n", "Hello\n# Header 1\nGoodbye\n", "

Hello

\n\n

Header 1

\n\n

Goodbye

\n", "* List\n# Header\n* List\n", "
    \n

  • List

    \n\n

    Header

    • \n\n

  • List

    • \n
\n", "* List\n#Header\n* List\n", "
    \n

  • List

    \n\n

    Header

    • \n\n

  • List

    • \n
\n", "* List\n * Nested list\n # Nested header\n", "
    \n

  • List

    \n\n
      \n

    • Nested list

      \n\n" + "

      Nested header

      • \n
    • \n
\n", } doTestsBlock(t, tests, 0) } func TestPrefixHeaderSpaceExtension(t *testing.T) { var tests = []string{ "# Header 1\n", "

Header 1

\n", "## Header 2\n", "

Header 2

\n", "### Header 3\n", "

Header 3

\n", "#### Header 4\n", "

Header 4

\n", "##### Header 5\n", "
Header 5
\n", "###### Header 6\n", "
Header 6
\n", "####### Header 7\n", "

####### Header 7

\n", "#Header 1\n", "

#Header 1

\n", "##Header 2\n", "

##Header 2

\n", "###Header 3\n", "

###Header 3

\n", "####Header 4\n", "

####Header 4

\n", "#####Header 5\n", "

#####Header 5

\n", "######Header 6\n", "

######Header 6

\n", "#######Header 7\n", "

#######Header 7

\n", "Hello\n# Header 1\nGoodbye\n", "

Hello

\n\n

Header 1

\n\n

Goodbye

\n", "* List\n# Header\n* List\n", "
    \n

  • List

    \n\n

    Header

    • \n\n

  • List

    • \n
\n", "* List\n#Header\n* List\n", "
    \n
  • List\n#Header
    • \n
  • List
    • \n
\n", "* List\n * Nested list\n # Nested header\n", "
    \n

  • List

    \n\n
      \n

    • Nested list

      \n\n" + "

      Nested header

      • \n
    • \n
\n", } doTestsBlock(t, tests, EXTENSION_SPACE_HEADERS) } func TestPrefixHeaderIdExtension(t *testing.T) { var tests = []string{ "# Header 1 {#someid}\n", "

Header 1

\n", "# Header 1 {#someid} \n", "

Header 1

\n", "# Header 1 {#someid}\n", "

Header 1

\n", "# Header 1 {#someid\n", "

Header 1 {#someid

\n", "# Header 1 {#someid\n", "

Header 1 {#someid

\n", "# Header 1 {#someid}}\n", "

Header 1

\n\n

}

\n", "## Header 2 {#someid}\n", "

Header 2

\n", "### Header 3 {#someid}\n", "

Header 3

\n", "#### Header 4 {#someid}\n", "

Header 4

\n", "##### Header 5 {#someid}\n", "
Header 5
\n", "###### Header 6 {#someid}\n", "
Header 6
\n", "####### Header 7 {#someid}\n", "
# Header 7
\n", "# Header 1 # {#someid}\n", "

Header 1

\n", "## Header 2 ## {#someid}\n", "

Header 2

\n", "Hello\n# Header 1\nGoodbye\n", "

Hello

\n\n

Header 1

\n\n

Goodbye

\n", "* List\n# Header {#someid}\n* List\n", "
    \n

  • List

    \n\n

    Header

    • \n\n

  • List

    • \n
\n", "* List\n#Header {#someid}\n* List\n", "
    \n

  • List

    \n\n

    Header

    • \n\n

  • List

    • \n
\n", "* List\n * Nested list\n # Nested header {#someid}\n", "
    \n

  • List

    \n\n
      \n

    • Nested list

      \n\n" + "

      Nested header

      • \n
    • \n
\n", } doTestsBlock(t, tests, EXTENSION_HEADER_IDS) } func TestPrefixAutoHeaderIdExtension(t *testing.T) { var tests = []string{ "# Header 1\n", "

Header 1

\n", "# Header 1 \n", "

Header 1

\n", "## Header 2\n", "

Header 2

\n", "### Header 3\n", "

Header 3

\n", "#### Header 4\n", "

Header 4

\n", "##### Header 5\n", "
Header 5
\n", "###### Header 6\n", "
Header 6
\n", "####### Header 7\n", "
# Header 7
\n", "Hello\n# Header 1\nGoodbye\n", "

Hello

\n\n

Header 1

\n\n

Goodbye

\n", "* List\n# Header\n* List\n", "
    \n

  • List

    \n\n

    Header

    • \n\n

  • List

    • \n
\n", "* List\n#Header\n* List\n", "
    \n

  • List

    \n\n

    Header

    • \n\n

  • List

    • \n
\n", "* List\n * Nested list\n # Nested header\n", "
    \n

  • List

    \n\n
      \n

    • Nested list

      \n\n" + "

      Nested header

      • \n
    • \n
\n", "# hallo\n\n # hallo\n\n # hallo\n\n # hallo\n", "

hallo

\n\n

hallo

\n\n

hallo

\n\n

hallo

\n", } doTestsBlock(t, tests, EXTENSION_AUTO_HEADER_IDS|EXTENSION_UNIQUE_HEADER_IDS) } func TestUnderlineHeaders(t *testing.T) { var tests = []string{ "Header 1\n========\n", "

Header 1

\n", "Header 2\n--------\n", "

Header 2

\n", "A\n=\n", "

A

\n", "B\n-\n", "

B

\n", "Paragraph\nHeader\n=\n", "

Paragraph

\n\n

Header

\n", "Header\n===\nParagraph\n", "

Header

\n\n

Paragraph

\n", "Header\n===\nAnother header\n---\n", "

Header

\n\n

Another header

\n", " Header\n======\n", "

Header

\n", " Code\n========\n", "
Code\n
\n\n

========

\n", "Header with *inline*\n=====\n", "

Header with inline

\n", "* List\n * Sublist\n Not a header\n ------\n", "
    \n
  • List\n\n
      \n
    • Sublist\nNot a header
      • \n
    \n\n
    • \n
\n", "Paragraph\n\n\n\n\nHeader\n===\n", "

Paragraph

\n\n

Header

\n", "Trailing space \n==== \n\n", "

Trailing space

\n", "Trailing spaces\n==== \n\n", "

Trailing spaces

\n", "Double underline\n=====\n=====\n", "

Double underline

\n\n

=====

\n", } doTestsBlock(t, tests, 0) } func TestUnderlineHeadersAutoIDs(t *testing.T) { var tests = []string{ "Header 1\n========\n", "

Header 1

\n", "Header 2\n--------\n", "

Header 2

\n", "A\n=\n", "

A

\n", "B\n-\n", "

B

\n", "Paragraph\nHeader\n=\n", "

Paragraph

\n\n

Header

\n", "Header\n===\nParagraph\n", "

Header

\n\n

Paragraph

\n", "Header\n===\nAnother header\n---\n", "

Header

\n\n

Another header

\n", " Header\n======\n", "

Header

\n", "Header with *inline*\n=====\n", "

Header with inline

\n", "Paragraph\n\n\n\n\nHeader\n===\n", "

Paragraph

\n\n

Header

\n", "Trailing space \n==== \n\n", "

Trailing space

\n", "Trailing spaces\n==== \n\n", "

Trailing spaces

\n", "Double underline\n=====\n=====\n", "

Double underline

\n\n

=====

\n", } doTestsBlock(t, tests, EXTENSION_AUTO_HEADER_IDS) } func TestHorizontalRule(t *testing.T) { var tests = []string{ "-\n", "

-

\n", "--\n", "

--

\n", "---\n", "
\n", "----\n", "
\n", "*\n", "

*

\n", "**\n", "

**

\n", "***\n", "
\n", "****\n", "
\n", "_\n", "

_

\n", "__\n", "

__

\n", "___\n", "
\n", "____\n", "
\n", "-*-\n", "

-*-

\n", "- - -\n", "
\n", "* * *\n", "
\n", "_ _ _\n", "
\n", "-----*\n", "

-----*

\n", " ------ \n", "
\n", "Hello\n***\n", "

Hello

\n\n
\n", "---\n***\n___\n", "
\n\n
\n\n
\n", } doTestsBlock(t, tests, 0) } func TestUnorderedList(t *testing.T) { var tests = []string{ "* Hello\n", "
    \n
  • Hello
    • \n
\n", "* Yin\n* Yang\n", "
    \n
  • Yin
    • \n
  • Yang
    • \n
\n", "* Ting\n* Bong\n* Goo\n", "
    \n
  • Ting
    • \n
  • Bong
    • \n
  • Goo
    • \n
\n", "* Yin\n\n* Yang\n", "
    \n

  • Yin

    • \n\n

  • Yang

    • \n
\n", "* Ting\n\n* Bong\n* Goo\n", "
    \n

  • Ting

    • \n\n

  • Bong

    • \n\n

  • Goo

    • \n
\n", "+ Hello\n", "
    \n
  • Hello
    • \n
\n", "+ Yin\n+ Yang\n", "
    \n
  • Yin
    • \n
  • Yang
    • \n
\n", "+ Ting\n+ Bong\n+ Goo\n", "
    \n
  • Ting
    • \n
  • Bong
    • \n
  • Goo
    • \n
\n", "+ Yin\n\n+ Yang\n", "
    \n

  • Yin

    • \n\n

  • Yang

    • \n
\n", "+ Ting\n\n+ Bong\n+ Goo\n", "
    \n

  • Ting

    • \n\n

  • Bong

    • \n\n

  • Goo

    • \n
\n", "- Hello\n", "
    \n
  • Hello
    • \n
\n", "- Yin\n- Yang\n", "
    \n
  • Yin
    • \n
  • Yang
    • \n
\n", "- Ting\n- Bong\n- Goo\n", "
    \n
  • Ting
    • \n
  • Bong
    • \n
  • Goo
    • \n
\n", "- Yin\n\n- Yang\n", "
    \n

  • Yin

    • \n\n

  • Yang

    • \n
\n", "- Ting\n\n- Bong\n- Goo\n", "
    \n

  • Ting

    • \n\n

  • Bong

    • \n\n

  • Goo

    • \n
\n", "*Hello\n", "

*Hello

\n", "* Hello \n", "
    \n
  • Hello
    • \n
\n", "* Hello \n Next line \n", "
    \n
  • Hello\nNext line
    • \n
\n", "Paragraph\n* No linebreak\n", "

Paragraph\n* No linebreak

\n", "Paragraph\n\n* Linebreak\n", "

Paragraph

\n\n
    \n
  • Linebreak
    • \n
\n", "* List\n * Nested list\n", "
    \n
  • List\n\n
      \n
    • Nested list
      • \n
    • \n
\n", "* List\n\n * Nested list\n", "
    \n

  • List

    \n\n
      \n
    • Nested list
      • \n
    • \n
\n", "* List\n Second line\n\n + Nested\n", "
    \n

  • List\nSecond line

    \n\n
      \n
    • Nested
      • \n
    • \n
\n", "* List\n + Nested\n\n Continued\n", "
    \n

  • List

    \n\n
      \n
    • Nested
      • \n
    \n\n

    Continued

    • \n
\n", "* List\n * shallow indent\n", "
    \n
  • List\n\n
      \n
    • shallow indent
      • \n
    • \n
\n", "* List\n" + " * shallow indent\n" + " * part of second list\n" + " * still second\n" + " * almost there\n" + " * third level\n", "
    \n" + "
  • List\n\n" + "
      \n" + "
    • shallow indent
      • \n" + "
    • part of second list
      • \n" + "
    • still second
      • \n" + "
    • almost there\n\n" + "
        \n" + "
      • third level
        • \n" + "
      • \n" + "
    • \n" + "
\n", "* List\n extra indent, same paragraph\n", "
    \n
  • List\n extra indent, same paragraph
    • \n
\n", "* List\n\n code block\n", "
    \n

  • List

    \n\n
    code block\n
    • \n
\n", "* List\n\n code block with spaces\n", "
    \n

  • List

    \n\n
      code block with spaces\n
    • \n
\n", "* List\n\n * sublist\n\n normal text\n\n * another sublist\n", "
    \n

  • List

    \n\n
      \n
    • sublist
      • \n
    \n\n

    normal text

    \n\n
      \n
    • another sublist
      • \n
    • \n
\n", `* Foo bar qux `, `

  • Foo

    bar

qux
`, } doTestsBlock(t, tests, 0) } func TestFencedCodeBlockWithinList(t *testing.T) { doTestsBlock(t, []string{ "* Foo\n\n ```\n bar\n\n qux\n ```\n", `

  • Foo

    bar

qux
`, }, EXTENSION_FENCED_CODE) } func TestOrderedList(t *testing.T) { var tests = []string{ "1. Hello\n", "
    \n
  1. Hello
    2. \n
\n", "1. Yin\n2. Yang\n", "
    \n
  1. Yin
    2. \n
  2. Yang
    3. \n
\n", "1. Ting\n2. Bong\n3. Goo\n", "
    \n
  1. Ting
    2. \n
  2. Bong
    3. \n
  3. Goo
    4. \n
\n", "1. Yin\n\n2. Yang\n", "
    \n

  1. Yin

    2. \n\n

  2. Yang

    3. \n
\n", "1. Ting\n\n2. Bong\n3. Goo\n", "
    \n

  1. Ting

    2. \n\n

  2. Bong

    3. \n\n

  3. Goo

    4. \n
\n", "1 Hello\n", "

1 Hello

\n", "1.Hello\n", "

1.Hello

\n", "1. Hello \n", "
    \n
  1. Hello
    2. \n
\n", "1. Hello \n Next line \n", "
    \n
  1. Hello\nNext line
    2. \n
\n", "Paragraph\n1. No linebreak\n", "

Paragraph\n1. No linebreak

\n", "Paragraph\n\n1. Linebreak\n", "

Paragraph

\n\n
    \n
  1. Linebreak
    2. \n
\n", "1. List\n 1. Nested list\n", "
    \n
  1. List\n\n
      \n
    1. Nested list
      2. \n
    2. \n
\n", "1. List\n\n 1. Nested list\n", "
    \n

  1. List

    \n\n
      \n
    1. Nested list
      2. \n
    2. \n
\n", "1. List\n Second line\n\n 1. Nested\n", "
    \n

  1. List\nSecond line

    \n\n
      \n
    1. Nested
      2. \n
    2. \n
\n", "1. List\n 1. Nested\n\n Continued\n", "
    \n

  1. List

    \n\n
      \n
    1. Nested
      2. \n
    \n\n

    Continued

    2. \n
\n", "1. List\n 1. shallow indent\n", "
    \n
  1. List\n\n
      \n
    1. shallow indent
      2. \n
    2. \n
\n", "1. List\n" + " 1. shallow indent\n" + " 2. part of second list\n" + " 3. still second\n" + " 4. almost there\n" + " 1. third level\n", "
    \n" + "
  1. List\n\n" + "
      \n" + "
    1. shallow indent
      2. \n" + "
    2. part of second list
      3. \n" + "
    3. still second
      4. \n" + "
    4. almost there\n\n" + "
        \n" + "
      1. third level
        2. \n" + "
      5. \n" + "
    2. \n" + "
\n", "1. List\n extra indent, same paragraph\n", "
    \n
  1. List\n extra indent, same paragraph
    2. \n
\n", "1. List\n\n code block\n", "
    \n

  1. List

    \n\n
    code block\n
    2. \n
\n", "1. List\n\n code block with spaces\n", "
    \n

  1. List

    \n\n
      code block with spaces\n
    2. \n
\n", "1. List\n * Mixted list\n", "
    \n
  1. List\n\n
      \n
    • Mixted list
      • \n
    2. \n
\n", "1. List\n * Mixed list\n", "
    \n
  1. List\n\n
      \n
    • Mixed list
      • \n
    2. \n
\n", "* Start with unordered\n 1. Ordered\n", "
    \n
  • Start with unordered\n\n
      \n
    1. Ordered
      2. \n
    • \n
\n", "* Start with unordered\n 1. Ordered\n", "
    \n
  • Start with unordered\n\n
      \n
    1. Ordered
      2. \n
    • \n
\n", "1. numbers\n1. are ignored\n", "
    \n
  1. numbers
    2. \n
  2. are ignored
    3. \n
\n", "a. hallo\n\na. item2\nb. item2\n", "

a. hallo

\n\n
    \n
  1. item2
    2. \n
  2. item2
    3. \n
\n", "i. hallo\n\ni. item2\nii. item2\n", "

i. hallo

\n\n
    \n
  1. item2
    2. \n
  2. item2
    3. \n
\n", "A. hallo\n\nA. item2\nB. item2\n", "

A. hallo

\n\n
    \n
  1. item2
    2. \n
  2. item2
    3. \n
\n", "1) item2\n2) item2\n", "
    \n
  1. item2
    2. \n
  2. item2
    3. \n
\n", "4. numbers\n1. are not ignored\n", "
    \n
  1. numbers
    2. \n
  2. are not ignored
    3. \n
\n", } doTestsBlock(t, tests, 0) } func TestAbbreviation(t *testing.T) { var tests = []string{ "*[HTML]: Hyper Text Markup Language\nHTML is cool", "

HTML is cool

\n", "*[HTML]:\nHTML is cool", "

HTML is cool

\n", "*[H]:\n cool H is", "

cool H is

\n", "*[H]:\n cool H is ", "

cool H is

\n", "*[H]: \n cool H is ", "

cool H is

\n", "*[H]: aa \n cool H is better yet some more words ", "

cool H is better yet some more words

\n", "*[H] aa \n cool H is ", "

*[H] aa
\n cool H is

\n", "*[HTML]: \"Hyper Text Markup Language\"\nHTML is cool", "

HTML is cool

\n", } doTestsBlock(t, tests, EXTENSION_ABBREVIATIONS) } func TestPreformattedHtml(t *testing.T) { var tests = []string{ "
\n", "
\n", "
\n
\n", "
\n
\n", "
\n
\nParagraph\n", "

\n
\nParagraph

\n", "
\n
\n", "
\n
\n", "
\nAnything here\n
\n", "
\nAnything here\n
\n", "
\n Anything here\n
\n", "
\n Anything here\n
\n", "
\nAnything here\n
\n", "
\nAnything here\n
\n", "
\nThis is *not* &proceessed\n
\n", "
\nThis is *not* &proceessed\n
\n", "\n Something\n\n", "

\n Something\n

\n", "
\n Something here\n\n", "

\n Something here\n

\n", "Paragraph\n
\nHere? >&<\n
\n", "

Paragraph\n

\nHere? >&<\n

\n", "Paragraph\n\n
\nHow about here? >&<\n
\n", "

Paragraph

\n\n
\nHow about here? >&<\n
\n", "Paragraph\n
\nHere? >&<\n
\nAnd here?\n", "

Paragraph\n

\nHere? >&<\n
\nAnd here?

\n", "Paragraph\n\n
\nHow about here? >&<\n
\nAnd here?\n", "

Paragraph

\n\n

\nHow about here? >&<\n
\nAnd here?

\n", "Paragraph\n
\nHere? >&<\n
\n\nAnd here?\n", "

Paragraph\n

\nHere? >&<\n

\n\n

And here?

\n", "Paragraph\n\n
\nHow about here? >&<\n
\n\nAnd here?\n", "

Paragraph

\n\n
\nHow about here? >&<\n
\n\n

And here?

\n", } doTestsBlock(t, tests, 0) } func TestPreformattedHtmlLax(t *testing.T) { var tests = []string{ "Paragraph\n
\nHere? >&<\n
\n", "

Paragraph

\n\n
\nHere? >&<\n
\n", "Paragraph\n\n
\nHow about here? >&<\n
\n", "

Paragraph

\n\n
\nHow about here? >&<\n
\n", "Paragraph\n
\nHere? >&<\n
\nAnd here?\n", "

Paragraph

\n\n
\nHere? >&<\n
\n\n

And here?

\n", "Paragraph\n\n
\nHow about here? >&<\n
\nAnd here?\n", "

Paragraph

\n\n
\nHow about here? >&<\n
\n\n

And here?

\n", "Paragraph\n
\nHere? >&<\n
\n\nAnd here?\n", "

Paragraph

\n\n
\nHere? >&<\n
\n\n

And here?

\n", "Paragraph\n\n
\nHow about here? >&<\n
\n\nAnd here?\n", "

Paragraph

\n\n
\nHow about here? >&<\n
\n\n

And here?

\n", } doTestsBlock(t, tests, EXTENSION_LAX_HTML_BLOCKS) } func TestFencedCodeBlock(t *testing.T) { var tests = []string{ "``` go\nfunc foo() bool {\n\treturn true;\n}\n```\n", "
func foo() bool {\n\treturn true;\n}\n
\n", "``` c\n/* special & char < > \" escaping */\n```\n", "
/* special & char < > " escaping */\n
\n", "``` c\nno *inline* processing ~~of text~~\n```\n", "
no *inline* processing ~~of text~~\n
\n", "```\nNo language\n```\n", "
No language\n
\n", "``` {ocaml}\nlanguage in braces\n```\n", "
language in braces\n
\n", "``` {ocaml} \nwith extra whitespace\n```\n", "
with extra whitespace\n
\n", "```{ ocaml }\nwith extra whitespace\n```\n", "
with extra whitespace\n
\n", "~ ~~ java\nWith whitespace\n~~~\n", "

~ ~~ java\nWith whitespace\n~~~

\n", "~~\nonly two\n~~\n", "

~~\nonly two\n~~

\n", "```` python\nextra\n````\n", "
extra\n
\n", "~~~ perl\nthree to start, four to end\n~~~~\n", "

~~~ perl\nthree to start, four to end\n~~~~

\n", "~~~~ perl\nfour to start, three to end\n~~~\n", "

~~~~ perl\nfour to start, three to end\n~~~

\n", "~~~ bash\ntildes\n~~~\n", "
tildes\n
\n", "``` lisp\nno ending\n", "

``` lisp\nno ending

\n", "~~~ lisp\nend with language\n~~~ lisp\n", "

~~~ lisp\nend with language\n~~~ lisp

\n", "```\nmismatched begin and end\n~~~\n", "

```\nmismatched begin and end\n~~~

\n", "~~~\nmismatched begin and end\n```\n", "

~~~\nmismatched begin and end\n```

\n", " ``` oz\nleading spaces\n```\n", "
leading spaces\n
\n", " ``` oz\nleading spaces\n ```\n", "
leading spaces\n
\n", " ``` oz\nleading spaces\n ```\n", "
leading spaces\n
\n", "``` oz\nleading spaces\n ```\n", "
leading spaces\n
\n", " ``` oz\nleading spaces\n ```\n", "
``` oz\n
\n\n

leading spaces\n ```

\n", "Bla bla\n\n``` oz\ncode blocks breakup paragraphs\n```\n\nBla Bla\n", "

Bla bla

\n\n
code blocks breakup paragraphs\n
\n\n

Bla Bla

\n", "Some text before a fenced code block\n``` oz\ncode blocks breakup paragraphs\n```\nAnd some text after a fenced code block", "

Some text before a fenced code block

\n\n
code blocks breakup paragraphs\n
\n\n

And some text after a fenced code block

\n", "`", "

`

\n", "Bla bla\n\n``` oz\ncode blocks breakup paragraphs\n```\n\nBla Bla\n\n``` oz\nmultiple code blocks work okay\n```\n\nBla Bla\n", "

Bla bla

\n\n
code blocks breakup paragraphs\n
\n\n

Bla Bla

\n\n
multiple code blocks work okay\n
\n\n

Bla Bla

\n", "Some text before a fenced code block\n``` oz\ncode blocks breakup paragraphs\n```\nSome text in between\n``` oz\nmultiple code blocks work okay\n```\nAnd some text after a fenced code block", "

Some text before a fenced code block

\n\n
code blocks breakup paragraphs\n
\n\n

Some text in between

\n\n
multiple code blocks work okay\n
\n\n

And some text after a fenced code block

\n", } doTestsBlock(t, tests, EXTENSION_FENCED_CODE) } func TestFencedCodeInsideBlockquotes(t *testing.T) { cat := func(s ...string) string { return strings.Join(s, "\n") } var tests = []string{ cat("> ```go", "package moo", "", "```", ""), `
package moo
`, // ------------------------------------------- cat("> foo", "> ", "> ```go", "package moo", "```", "> ", "> goo.", ""), `

foo

package moo

goo.

`, // ------------------------------------------- cat("> foo", "> ", "> quote", "continues", "```", ""), `

foo

quote continues ` + "```" + `

`, // ------------------------------------------- cat("> foo", "> ", "> ```go", "package moo", "```", "> ", "> goo.", "> ", "> ```go", "package zoo", "```", "> ", "> woo.", ""), `

foo

package moo

goo.

package zoo

woo.

`, } // These 2 alternative forms of blockquoted fenced code blocks should produce same output. forms := [2]string{ cat("> plain quoted text", "> ```fenced", "code", " with leading single space correctly preserved", "okay", "```", "> rest of quoted text"), cat("> plain quoted text", "> ```fenced", "> code", "> with leading single space correctly preserved", "> okay", "> ```", "> rest of quoted text"), } want := `

plain quoted text

code
 with leading single space correctly preserved
okay

rest of quoted text

` tests = append(tests, forms[0], want) tests = append(tests, forms[1], want) doTestsBlock(t, tests, EXTENSION_FENCED_CODE) } func TestTable(t *testing.T) { var tests = []string{ "a | b\n---|---\nc | d\n", "\n\n\n\n\n\n\n\n" + "\n\n\n\n\n\n
ab
cd
\n", "a | b\n---|--\nc | d\n", "

a | b\n---|--\nc | d

\n", "|a|b|c|d|\n|----|----|----|---|\n|e|f|g|h|\n", "\n\n\n\n\n\n\n\n\n\n" + "\n\n\n\n\n\n\n\n
abcd
efgh
\n", "*a*|__b__|[c](C)|d\n---|---|---|---\ne|f|g|h\n", "\n\n\n\n\n\n\n\n\n\n" + "\n\n\n\n\n\n\n\n
abcd
efgh
\n", "a|b|c\n---|---|---\nd|e|f\ng|h\ni|j|k|l|m\nn|o|p\n", "\n\n\n\n\n\n\n\n\n" + "\n\n\n\n\n\n\n" + "\n\n\n\n\n\n" + "\n\n\n\n\n\n" + "\n\n\n\n\n\n
abc
def
gh
ijk
nop
\n", "a|b|c\n---|---|---\n*d*|__e__|f\n", "\n\n\n\n\n\n\n\n\n" + "\n\n\n\n\n\n\n
abc
def
\n", "a|b|c|d\n:--|--:|:-:|---\ne|f|g|h\n", "\n\n\n\n\n" + "\n\n\n\n\n" + "\n\n\n\n" + "\n\n\n\n
abcd
efgh
\n", "a|b|c\n---|---|---\n", "\n\n\n\n\n\n\n\n\n\n\n
abc
\n", "a| b|c | d | e\n---|---|---|---|---\nf| g|h | i |j\n", "\n\n\n\n\n\n\n\n\n\n\n" + "\n\n\n\n\n\n\n\n\n
abcde
fghij
\n", "a|b\\|c|d\n---|---|---\nf|g\\|h|i\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ab|cd
fg|hi
\n", "a | c\n--- | ---:\nd | e\nf | g\n== | =\nh | j\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ac
de
fg
hj
\n", "a | c\n--- | --:\nd | e\nf | g\n== | =\n== | =\nh | j\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ac
de
fg
===
hj
\n", } doTestsBlock(t, tests, EXTENSION_TABLES) } func TestBlockTable(t *testing.T) { var tests = []string{ "|--------+--------+\n| Defaul |Left ald|\n|--------|--------|\n| Second |foo |\n|--------+--------+\n| Second | 2. Ite |\n| 2 line | 3. Ite |\n|--------+--------+\n| Footer | Footer |\n|--------+--------+\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
DefaulLeft ald

Second

\n

foo

\n

Second\n2 line

\n
    \n
  1. Ite
    2. \n
  2. Ite
    3. \n
\n

Footer

\n

Footer

\n
\n", "|--------+--------+\n| Defaul |Left ald|\n|--------|--------|\n| Second |foo |\n|--------+--------+\n| Second | 2. Ite |\n| 2 line | 3. Ite |\n|--------+--------+\n| Footer | Footer |\n|--------+--------+\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
DefaulLeft ald

Second

\n

foo

\n

Second\n2 line

\n
    \n
  1. Ite
    2. \n
  2. Ite
    3. \n
\n

Footer

\n

Footer

\n
\n", "|--------+--------+\n| Defaul |Left ald|\n|--------|--------|\n| Second |foo |\n|--------+--------+\n| Second | 2. Ite |\n| 2 line | 3. Ite |\n|--------+--------+\n| Footer | Footer |\n|--------+--------+\nTable: this is a table\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\nthis is a table\n
DefaulLeft ald

Second

\n

foo

\n

Second\n2 line

\n
    \n
  1. Ite
    2. \n
  2. Ite
    3. \n
\n

Footer

\n

Footer

\n
\n", "|--------+--------+\n| Defaul |Left ald|\n|--------|--------|\n| Second |foo |\n|--------+--------+\n| Second | 2. Ite |\n| 2 line | 3. Ite |\n+========+========+\n| Footer | Footer |\n|--------+--------+\nTable: this is a table\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\nthis is a table\n
DefaulLeft ald

Second

\n

foo

\n

Second\n2 line

\n
    \n
  1. Ite
    2. \n
  2. Ite
    3. \n
\n
FooterFooter
\n", "|--------+--------+\n|--------+--------+\n| Defaul |Left ald|\n|--------|--------|\n|--------+--------+\n| Second | 2. Ite |\n| 2 line | 3. Ite |\n+========+========+\n|--------+--------+\nTable: this is a table\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\nthis is a table\n

Defaul

\n

Second\n2 line

\n
\n", "|--------+--------+\n|--------+--------+\n| Defaul |Left ald|\n|--------|--------|\n|--------+--------+\n| Second | 2. Ite |\n| 2 line | 3. Ite |\n+========+========+\n+========+========+\n|--------+--------+\nTable: this is a table\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Defaul

\n

Second\n2 line

\n
\n\n

+========+========+\n|--------+--------+\nTable: this is a table

\n", "+--------+--------+\n| Defaul |Left ald|\n|--------|--------|\n|--------+--------+\n| Second | 2. Ite |\n| 2 line | 3. Ite |\n+========+========+\n|--------+--------+\nTable: this is a table\n", "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\nthis is a table\n
DefaulLeft ald

Second\n2 line

\n
    \n
  1. Ite
    2. \n
  2. Ite
    3. \n
\n
\n", "--\n**\n--\n", "

--

\n\n

**

\n", } doTestsBlock(t, tests, EXTENSION_TABLES) } func TestUnorderedListWith_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) { var tests = []string{ "* Hello\n", "
    \n
  • Hello
    • \n
\n", "* Yin\n* Yang\n", "
    \n
  • Yin
    • \n
  • Yang
    • \n
\n", "* Ting\n* Bong\n* Goo\n", "
    \n
  • Ting
    • \n
  • Bong
    • \n
  • Goo
    • \n
\n", "* Yin\n\n* Yang\n", "
    \n

  • Yin

    • \n\n

  • Yang

    • \n
\n", "* Ting\n\n* Bong\n* Goo\n", "
    \n

  • Ting

    • \n\n

  • Bong

    • \n\n

  • Goo

    • \n
\n", "+ Hello\n", "
    \n
  • Hello
    • \n
\n", "+ Yin\n+ Yang\n", "
    \n
  • Yin
    • \n
  • Yang
    • \n
\n", "+ Ting\n+ Bong\n+ Goo\n", "
    \n
  • Ting
    • \n
  • Bong
    • \n
  • Goo
    • \n
\n", "+ Yin\n\n+ Yang\n", "
    \n

  • Yin

    • \n\n

  • Yang

    • \n
\n", "+ Ting\n\n+ Bong\n+ Goo\n", "
    \n

  • Ting

    • \n\n

  • Bong

    • \n\n

  • Goo

    • \n
\n", "- Hello\n", "
    \n
  • Hello
    • \n
\n", "- Yin\n- Yang\n", "
    \n
  • Yin
    • \n
  • Yang
    • \n
\n", "- Ting\n- Bong\n- Goo\n", "
    \n
  • Ting
    • \n
  • Bong
    • \n
  • Goo
    • \n
\n", "- Yin\n\n- Yang\n", "
    \n

  • Yin

    • \n\n

  • Yang

    • \n
\n", "- Ting\n\n- Bong\n- Goo\n", "
    \n

  • Ting

    • \n\n

  • Bong

    • \n\n

  • Goo

    • \n
\n", "*Hello\n", "

*Hello

\n", "* Hello \n", "
    \n
  • Hello
    • \n
\n", "* Hello \n Next line \n", "
    \n
  • Hello\nNext line
    • \n
\n", "Paragraph\n* No linebreak\n", "

Paragraph

\n\n
    \n
  • No linebreak
    • \n
\n", "Paragraph\n\n* Linebreak\n", "

Paragraph

\n\n
    \n
  • Linebreak
    • \n
\n", "* List\n * Nested list\n", "
    \n
  • List\n\n
      \n
    • Nested list
      • \n
    • \n
\n", "* List\n\n * Nested list\n", "
    \n

  • List

    \n\n
      \n
    • Nested list
      • \n
    • \n
\n", "* List\n Second line\n\n + Nested\n", "
    \n

  • List\nSecond line

    \n\n
      \n
    • Nested
      • \n
    • \n
\n", "* List\n + Nested\n\n Continued\n", "
    \n

  • List

    \n\n
      \n
    • Nested
      • \n
    \n\n

    Continued

    • \n
\n", "* List\n * shallow indent\n", "
    \n
  • List\n\n
      \n
    • shallow indent
      • \n
    • \n
\n", "* List\n" + " * shallow indent\n" + " * part of second list\n" + " * still second\n" + " * almost there\n" + " * third level\n", "
    \n" + "
  • List\n\n" + "
      \n" + "
    • shallow indent
      • \n" + "
    • part of second list
      • \n" + "
    • still second
      • \n" + "
    • almost there\n\n" + "
        \n" + "
      • third level
        • \n" + "
      • \n" + "
    • \n" + "
\n", "* List\n extra indent, same paragraph\n", "
    \n
  • List\n extra indent, same paragraph
    • \n
\n", "* List\n\n code block\n", "
    \n

  • List

    \n\n
    code block\n
    • \n
\n", "* List\n\n code block with spaces\n", "
    \n

  • List

    \n\n
      code block with spaces\n
    • \n
\n", "* List\n\n * sublist\n\n normal text\n\n * another sublist\n", "
    \n

  • List

    \n\n
      \n
    • sublist
      • \n
    \n\n

    normal text

    \n\n
      \n
    • another sublist
      • \n
    • \n
\n", } doTestsBlock(t, tests, EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK) } func TestOrderedList_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) { var tests = []string{ "1. Hello\n", "
    \n
  1. Hello
    2. \n
\n", "1. Yin\n2. Yang\n", "
    \n
  1. Yin
    2. \n
  2. Yang
    3. \n
\n", "1. Ting\n2. Bong\n3. Goo\n", "
    \n
  1. Ting
    2. \n
  2. Bong
    3. \n
  3. Goo
    4. \n
\n", "1. Yin\n\n2. Yang\n", "
    \n

  1. Yin

    2. \n\n

  2. Yang

    3. \n
\n", "1. Ting\n\n2. Bong\n3. Goo\n", "
    \n

  1. Ting

    2. \n\n

  2. Bong

    3. \n\n

  3. Goo

    4. \n
\n", "1 Hello\n", "

1 Hello

\n", "1.Hello\n", "

1.Hello

\n", "1. Hello \n", "
    \n
  1. Hello
    2. \n
\n", "1. Hello \n Next line \n", "
    \n
  1. Hello\nNext line
    2. \n
\n", "Paragraph\n1. No linebreak\n", "

Paragraph

\n\n
    \n
  1. No linebreak
    2. \n
\n", "Paragraph\n\n1. Linebreak\n", "

Paragraph

\n\n
    \n
  1. Linebreak
    2. \n
\n", "1. List\n 1. Nested list\n", "
    \n
  1. List\n\n
      \n
    1. Nested list
      2. \n
    2. \n
\n", "1. List\n\n 1. Nested list\n", "
    \n

  1. List

    \n\n
      \n
    1. Nested list
      2. \n
    2. \n
\n", "1. List\n Second line\n\n 1. Nested\n", "
    \n

  1. List\nSecond line

    \n\n
      \n
    1. Nested
      2. \n
    2. \n
\n", "1. List\n 1. Nested\n\n Continued\n", "
    \n

  1. List

    \n\n
      \n
    1. Nested
      2. \n
    \n\n

    Continued

    2. \n
\n", "1. List\n 1. shallow indent\n", "
    \n
  1. List\n\n
      \n
    1. shallow indent
      2. \n
    2. \n
\n", "1. List\n" + " 1. shallow indent\n" + " 2. part of second list\n" + " 3. still second\n" + " 4. almost there\n" + " 1. third level\n", "
    \n" + "
  1. List\n\n" + "
      \n" + "
    1. shallow indent
      2. \n" + "
    2. part of second list
      3. \n" + "
    3. still second
      4. \n" + "
    4. almost there\n\n" + "
        \n" + "
      1. third level
        2. \n" + "
      5. \n" + "
    2. \n" + "
\n", "1. List\n extra indent, same paragraph\n", "
    \n
  1. List\n extra indent, same paragraph
    2. \n
\n", "1. List\n\n code block\n", "
    \n

  1. List

    \n\n
    code block\n
    2. \n
\n", "1. List\n\n code block with spaces\n", "
    \n

  1. List

    \n\n
      code block with spaces\n
    2. \n
\n", "1. List\n * Mixted list\n", "
    \n
  1. List\n\n
      \n
    • Mixted list
      • \n
    2. \n
\n", "1. List\n * Mixed list\n", "
    \n
  1. List\n\n
      \n
    • Mixed list
      • \n
    2. \n
\n", "* Start with unordered\n 1. Ordered\n", "
    \n
  • Start with unordered\n\n
      \n
    1. Ordered
      2. \n
    • \n
\n", "* Start with unordered\n 1. Ordered\n", "
    \n
  • Start with unordered\n\n
      \n
    1. Ordered
      2. \n
    • \n
\n", "1. numbers\n1. are ignored\n", "
    \n
  1. numbers
    2. \n
  2. are ignored
    3. \n
\n", "a. List\nb. Item\n", "
    \n
  1. List
    2. \n
  2. Item
    3. \n
\n", "aa. List\nbb. Item\n", "
    \n
  1. List
    2. \n
  2. Item
    3. \n
\n", "aaa. List\aab. Item\n", "

aaa. List\aab. Item

\n", } doTestsBlock(t, tests, EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK) } func TestFencedCodeBlock_EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK(t *testing.T) { var tests = []string{ "``` go\nfunc foo() bool {\n\treturn true;\n}\n```\n", "
func foo() bool {\n\treturn true;\n}\n
\n", "``` c\n/* special & char < > \" escaping */\n```\n", "
/* special & char < > " escaping */\n
\n", "``` c\nno *inline* processing ~~of text~~\n```\n", "
no *inline* processing ~~of text~~\n
\n", "```\nNo language\n```\n", "
No language\n
\n", "``` {ocaml}\nlanguage in braces\n```\n", "
language in braces\n
\n", "``` {ocaml} \nwith extra whitespace\n```\n", "
with extra whitespace\n
\n", "```{ ocaml }\nwith extra whitespace\n```\n", "
with extra whitespace\n
\n", "~ ~~ java\nWith whitespace\n~~~\n", "

~ ~~ java\nWith whitespace\n~~~

\n", "~~\nonly two\n~~\n", "

~~\nonly two\n~~

\n", "```` python\nextra\n````\n", "
extra\n
\n", "~~~ perl\nthree to start, four to end\n~~~~\n", "

~~~ perl\nthree to start, four to end\n~~~~

\n", "~~~~ perl\nfour to start, three to end\n~~~\n", "

~~~~ perl\nfour to start, three to end\n~~~

\n", "~~~ bash\ntildes\n~~~\n", "
tildes\n
\n", "``` lisp\nno ending\n", "

``` lisp\nno ending

\n", "~~~ lisp\nend with language\n~~~ lisp\n", "

~~~ lisp\nend with language\n~~~ lisp

\n", "```\nmismatched begin and end\n~~~\n", "

```\nmismatched begin and end\n~~~

\n", "~~~\nmismatched begin and end\n```\n", "

~~~\nmismatched begin and end\n```

\n", " ``` oz\nleading spaces\n```\n", "
leading spaces\n
\n", " ``` oz\nleading spaces\n ```\n", "
leading spaces\n
\n", " ``` oz\nleading spaces\n ```\n", "
leading spaces\n
\n", "``` oz\nleading spaces\n ```\n", "
leading spaces\n
\n", " ``` oz\nleading spaces\n ```\n", "
``` oz\n
\n\n

leading spaces

\n\n
```\n
\n", } doTestsBlock(t, tests, EXTENSION_FENCED_CODE|EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK) } func TestCDATA(t *testing.T) { var tests = []string{ "Some text\n\n\n", "

Some text

\n\n\n", "CDATA ]]\n\n\n", "

CDATA ]]

\n\n\n", "CDATA >\n\n]]>\n", "

CDATA >

\n\n]]>\n", "Lots of text\n\n\n", "

Lots of text

\n\n\n", "]]>\n", "]]>\n", } doTestsBlock(t, tests, 0) doTestsBlock(t, []string{ "``` html\n\n```\n", "
<![CDATA[foo]]>\n
\n", "\n", "\n", ` def func(): > pass ]]> `, ` def func(): > pass ]]> `, }, EXTENSION_FENCED_CODE) } func TestDefinitionListXML(t *testing.T) { var tests = []string{ "Term1\n: Hi There", "
\n
Term1
\n
Hi There
\n
\n", "Term1\n: Yin\nTerm2\n: Yang\n", "
\n
Term1
\n
Yin
\n
Term2
\n
Yang
\n
\n", // fix sourcecode/artwork here. `Term 1 : This is a definition with two paragraphs. Lorem ipsum Vestibulum enim wisi, viverra nec, fringilla in, laoreet vitae, risus. Term 2 : This definition has a code block, a blockquote and a list. code block. > block quote > on two lines. 1. first list item 2. second list item `, "
\n
Term 1
\n
\nThis is a definition with two paragraphs. Lorem ipsum\n\n\nVestibulum enim wisi, viverra nec, fringilla in, laoreet\nvitae, risus.\n
\n
Term 2
\n
\nThis definition has a code block, a blockquote and a list.\n\n\ncode block.\n\n
\n\nblock quote\non two lines.\n\n
\n
    \n
  1. first list item
    2. \n
  2. second list item
    3. \n
\n
\n", `Apple : Pomaceous fruit of plants of the genus Malus in the family Rosaceae. Orange and *Apples* : The thing of an evergreen tree of the genus Citrus.`, "
\n
Apple
\n
\nPomaceous fruit of plants of the genus Malus in\nthe family Rosaceae.\n
\n
Orange and Apples
\n
\nThe thing of an evergreen tree of the genus Citrus.\n
\n
\n", `Apple : Pomaceous fruit of plants of the genus Malus in the family Rosaceae. Orange and *Apples* : The thing of an evergreen tree of the genus Citrus.`, "
\n
Apple
\n
\nPomaceous fruit of plants of the genus Malus in\nthe family Rosaceae.\n
\n
Orange and Apples
\n
\nThe thing of an evergreen tree of the genus Citrus.\n
\n
\n", } doTestsBlockXML(t, tests, 0) } func TestAbstractNoteAsideXML(t *testing.T) { var tests = []string{ ".# Abstract\nbegin of abstract\n\nthis is an abstract\n", "\n\n\nbegin of abstract\n\n\nthis is an abstract\n\n\n\n", ".# A note\nthis is the content\n", "\n\nA note\n\nthis is the content\n\n\n\n", "A> begin of aside\nA> this is an aside\n", "\n", } doTestsBlockXML(t, tests, 0) } func TestOrderedListStartXML(t *testing.T) { var tests = []string{ "1. hello\n1. hello\n\ndivide\n\n4. hello\n5. hello\n\ndivide\n\n 7. hello\n5. hello\n", "
    \n
  1. hello
    2. \n
  2. hello
    3. \n
\n\ndivide\n\n
    \n
  1. hello
    2. \n
  2. hello
    3. \n
\n\ndivide\n\n
    \n
  1. hello
    2. \n
  2. hello
    3. \n
\n", } doTestsBlockXML(t, tests, 0) } func TestIncludesXML(t *testing.T) { if !testing.Short() { return } var tests = []string{ "{{/dev/null}}", "", "<{{/dev/null}}", "\n\n\n\n\n\n", " <{{/dev/null}}", "\n\n\n\n\n\n", "`{{does-not-exist}}`", "\n{{does-not-exist}}\n\n", " {{does-not-exist}}", "\n{{does-not-exist}}\n\n", "`<{{prog-not-exist}}`", "\n<{{prog-not-exist}}\n\n", `1. This is item1 2. This is item2. Lets include some code: <{{/dev/null}} Figure: this is some code!`, "
    \n
  1. This is item1
    2. \n
  2. This is item2.\nLets include some code:\n
    \nthis is some code!\n\n\n\n
    3. \n
\n", } f, e := ioutil.TempFile("/tmp", "mmark_test.") if e == nil { defer os.Remove(f.Name()) ioutil.WriteFile(f.Name(), []byte(` tedious_code = boring_function() // START OMIT interesting_code = fascinating_function() // END OMIT`), 0644) t1 := "Include some code\n <{{" + f.Name() + "}}[/START OMIT/,/END OMIT/]\n" e1 := "\nInclude some code\n \ninteresting_code = fascinating_function()\n\n\n\n" tests = append(tests, []string{t1, e1}...) } doTestsBlockXML(t, tests, EXTENSION_INCLUDE) } func TestInlineAttrXML(t *testing.T) { var tests = []string{ "{attribution=\"BLA BLA\" .green}\n{bla=BLA}\n{more=\"ALB ALB\" #ref:quote .yellow}\n> Hallo2\n> Hallo3\n\nThis is no one `{source='BLEIP'}` on of them\n\n{evenmore=\"BLE BLE\"}\n> Hallo6\n> Hallo7", "
\n\nHallo2\nHallo3\n\n
\n\nThis is no one {source='BLEIP'} on of them\n\n
\n\nHallo6\nHallo7\n\n
\n", "{style=\"format REQ(%c)\" start=\"4\"}\n1. Term1\n2. Term2", "
    \n
  1. Term1
    2. \n
  2. Term2
    3. \n
\n", " {style=\"format REQ(%c)\" start=\"4\"}\n1. Term1\n2. Term2", "\n{style=\"format REQ(%c)\" start=\"4\"}\n\n
    \n
  1. Term1
    2. \n
  2. Term2
    3. \n
\n", "{.green #ref1}\n# hallo\n\n{.yellow}\n# hallo {#ref2}\n\n{.blue #ref3}\n# hallo {#ref4}\n", "\n
\nhallo\n
\n\n
\nhallo\n
\n\n
\nhallo\n
\n", } doTestsBlockXML(t, tests, 0) } func TestCloseHeaderXML(t *testing.T) { var tests = []string{ "# Header1\n", "\n
\nHeader1\n
\n", } doTestsBlockXML(t, tests, 0) } func TestOrderedExampleListXML(t *testing.T) { var tests = []string{ `(@good) Example1 (@good) Example2 As this illustrates (@good) Example3 As we can see from (@good) some things never change. Some we do not when we reference (@good1). `, "
    \n
  1. \nExample1\n
    2. \n
  2. \nExample2\n
    3. \n
\n\nAs this illustrates\n\n
    \n
  1. Example3
    2. \n
\n\nAs we can see from (2) some things never change. Some we do not when we reference (@good1).\n\n", } doTestsBlockXML(t, tests, 0) } // TODO(miek): titleblock, this will only work with full document output. func testTitleBlockTOML(t *testing.T) { var tests = []string{ `% Title = "Test Document" % abbr = "test" {mainmatter} `, "dsjsdk", } doTestsBlockXML(t, tests, EXTENSION_TITLEBLOCK_TOML) } func TestSubFiguresXML(t *testing.T) { var tests = []string{` * Item1 * Item2 Basic usage: F> {type="ascii-art"} F> +-----+ F> | ART | F> +-----+ F> Figure: This caption is ignored. F> F> ~~~ c F> printf("%s\n", "hello"); F> ~~~ Figure: Caption you will see, for both figures. `, "
    \n
  • Item1
    • \n
  • \nItem2\n\n\nBasic usage:\n\n
    \nCaption you will see, for both figures.\n\n\n\n\n +-----+\n | ART |\n +-----+\n\n\n\nprintf(\"%s\\n\", \"hello\");\n\n
    • \n
\n", ` And another one F> {type="ascii-art"} F> +-----+ F> | ART | F> +-----+ F> Figure: This caption is ignored. F> F> ~~~ c F> printf("%s\n", "hello"); F> ~~~ F> Figure: Caption you will see, for both figures.`, "\nAnd another one\n\n
\nCaption you will see, for both figures.\n\n\n\n\n +-----+\n | ART |\n +-----+\n\n\n\nprintf(\"%s\\n\", \"hello\");\n\n
\n", } doTestsBlockXML(t, tests, 0) } func TestBlockComments(t *testing.T) { var tests = []string{ "Some text\n\n\n", "

Some text

\n\n\n", "Some text\n\n\n", "

Some text

\n\n\n", "Some text\n\n\n", "

Some text

\n\n\n", } doTestsBlock(t, tests, 0) } func TestIALXML(t *testing.T) { var tests = []string{ "{#id}\n Code\n", "\nCode\n\n", "{id}\n Code\n", "\n{id}\n\n\nCode\n\n", "{#id}\n", "", "{#id}\n{type=\"go\"}\n Code\n", "\nCode\n\n", "{#id \n Code\n", "\n{#id\n\n\nCode\n\n", } doTestsBlockXML(t, tests, 0) } func testCalloutXML(t *testing.T) { var tests = []string{` {callout="true"} This is some code Code <1> More <1> Not a callout \<3> As you can see in <1> we do some funky stuff above here. `, ""} doTestsBlockXML(t, tests, 0) } // TODO: // figure caption // table caption // frontmatter mmark-1.3-git849458a/cm_test.go000066400000000000000000000047561264073172300161220ustar00rootroot00000000000000// Unit tests for commonmark parsing package mmark import ( "testing" ) func runMarkdownCommonMark(input string, extensions int) string { htmlFlags := 0 renderer := HtmlRenderer(htmlFlags, "", "") return Parse([]byte(input), renderer, extensions).String() } func doTestsCommonMark(t *testing.T, tests []string, extensions int) { // catch and report panics var candidate string defer func() { if err := recover(); err != nil { t.Errorf("\npanic while processing [%#v]: %s\n", candidate, err) } }() for i := 0; i+1 < len(tests); i += 2 { input := tests[i] candidate = input expected := tests[i+1] actual := runMarkdownCommonMark(candidate, extensions) if actual != expected { t.Errorf("\nInput [%#v]\nExpected[%#v]\nActual [%#v]", candidate, expected, actual) } // now test every substring to stress test bounds checking if !testing.Short() { for start := 0; start < len(input); start++ { for end := start + 1; end <= len(input); end++ { candidate = input[start:end] _ = runMarkdownBlock(candidate, extensions) } } } } } func TestPrefixHeaderCommonMark_29(t *testing.T) { var tests = []string{ "# hallo\n\n # hallo\n\n # hallo\n\n # hallo\n\n # hallo\n", "

hallo

\n\n

hallo

\n\n

hallo

\n\n

hallo

\n\n
# hallo\n
\n", } doTestsCommonMark(t, tests, 0) } func TestHRuleCommonMark_18_22(t *testing.T) { var tests = []string{ "* List\n * Sublist\n Not a header\n ------\n", "
    \n
  • List\n\n
      \n
    • Sublist\nNot a header
      • \n
    \n\n
    • \n
\n", "- Foo\n- * * *\n- Bar\n", "
    \n
  • Foo
    • \n
  • * * *
    • \n
  • Bar
    • \n
\n", "* Foo\n* * * *\n", "
    \n
  • Foo
    • \n
\n\n
\n", "* Foo\n * - - -\n", "
    \n
  • Foo\n\n
      \n
    • - - -
      • \n
    • \n
\n", } doTestsCommonMark(t, tests, 0) } func TestFencedCodeBlockCommonMark_81(t *testing.T) { var tests = []string{ " ```\n aaa\naaa\n```\n", "
aaa\naaa\n
\n", "```\n bbb\nbbb\n```\n", "
 bbb\nbbb\n
\n", // disabled this behavior, using blackfriday's standard now. "```\nbbb\nbbb\n", //"
bbb\nbbb\n
\n", "

```\nbbb\nbbb

\n", "~~~~ ruby\ndef foo(x)\n return 3\nend\n~~~~\n", "
def foo(x)\n return 3\nend\n
\n", "```\n[foo]: /url\n```\n", "
[foo]: /url\n
\n", } doTestsCommonMark(t, tests, EXTENSION_FENCED_CODE) } mmark-1.3-git849458a/code.go000066400000000000000000000156271264073172300153750ustar00rootroot00000000000000// Copyright 2012 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. // Adapted for mmark, by Miek Gieben, 2015. package mmark import ( "bufio" "bytes" "errors" "io/ioutil" "regexp" "strconv" "unicode/utf8" ) // SourceCodeTypes are the different languages that are supported as // a type attribute in sourcecode, see Section 2.48.4 of XML2RFC v3 (-21). var SourceCodeTypes = map[string]bool{ "abnf": true, "asn.1": true, "bash": true, "c++": true, "c": true, "cbor": true, "dtd": true, "java": true, "javascript": true, "json": true, "mib": true, "perl": true, "pseudocode": true, "python": true, "rnc": true, "xml": true, "go": true, } // parseCode parses a code address directive. func parseCode(addr []byte, file []byte) []byte { bytes.TrimSpace(addr) textBytes, err := ioutil.ReadFile(string(file)) if err != nil { printf(nil, "failed: `%s': %s", string(file), err) return nil } lo, hi, err := addrToByteRange(string(addr), 0, textBytes) if err != nil { printf(nil, "code include address: %s", err.Error()) return textBytes } // Acme pattern matches can stop mid-line, // so run to end of line in both directions if not at line start/end. for lo > 0 && textBytes[lo-1] != '\n' { lo-- } if hi > 0 { for hi < len(textBytes) && textBytes[hi-1] != '\n' { hi++ } } lines := codeLines(textBytes, lo, hi) return lines } // codeLines takes a source file and returns the lines that // span the byte range specified by start and end. // It discards lines that end in "OMIT" and in "OMIT -->" func codeLines(src []byte, start, end int) (lines []byte) { startLine := 1 for i, b := range src { if i == start { break } if b == '\n' { startLine++ } } s := bufio.NewScanner(bytes.NewReader(src[start:end])) for n := startLine; s.Scan(); n++ { l := s.Bytes() if bytes.HasSuffix(l, []byte("OMIT")) { continue } if bytes.HasSuffix(l, []byte("OMIT -->")) { continue } lines = append(lines, l...) lines = append(lines, '\n') } // TODO(miek): trim leading and trailing blanklines return } // This file is stolen from go/src/cmd/godoc/codewalk.go. // It's an evaluator for the file address syntax implemented by acme and sam, // but using Go-native regular expressions. // To keep things reasonably close, this version uses (?m:re) for all user-provided // regular expressions. That is the only change to the code from codewalk.go. // See http://plan9.bell-labs.com/sys/doc/sam/sam.html Table II // for details on the syntax. // addrToByte evaluates the given address starting at offset start in data. // It returns the lo and hi byte offset of the matched region within data. func addrToByteRange(addr string, start int, data []byte) (lo, hi int, err error) { if addr == "" { lo, hi = start, len(data) return } var ( dir byte prevc byte charOffset bool ) lo = start hi = start for addr != "" && err == nil { c := addr[0] switch c { default: err = errors.New("invalid address syntax near " + string(c)) case ',': if len(addr) == 1 { hi = len(data) } else { _, hi, err = addrToByteRange(addr[1:], hi, data) } return case '+', '-': if prevc == '+' || prevc == '-' { lo, hi, err = addrNumber(data, lo, hi, prevc, 1, charOffset) } dir = c case '$': lo = len(data) hi = len(data) if len(addr) > 1 { dir = '+' } case '#': charOffset = true case '0', '1', '2', '3', '4', '5', '6', '7', '8', '9': var i int for i = 1; i < len(addr); i++ { if addr[i] < '0' || addr[i] > '9' { break } } var n int n, err = strconv.Atoi(addr[0:i]) if err != nil { break } lo, hi, err = addrNumber(data, lo, hi, dir, n, charOffset) dir = 0 charOffset = false prevc = c addr = addr[i:] continue case '/': var i, j int Regexp: for i = 1; i < len(addr); i++ { switch addr[i] { case '\\': i++ case '/': j = i + 1 break Regexp } } if j == 0 { j = i } pattern := addr[1:i] lo, hi, err = addrRegexp(data, lo, hi, dir, pattern) prevc = c addr = addr[j:] continue } prevc = c addr = addr[1:] } if err == nil && dir != 0 { lo, hi, err = addrNumber(data, lo, hi, dir, 1, charOffset) } if err != nil { return 0, 0, err } return lo, hi, nil } // addrNumber applies the given dir, n, and charOffset to the address lo, hi. // dir is '+' or '-', n is the count, and charOffset is true if the syntax // used was #n. Applying +n (or +#n) means to advance n lines // (or characters) after hi. Applying -n (or -#n) means to back up n lines // (or characters) before lo. // The return value is the new lo, hi. func addrNumber(data []byte, lo, hi int, dir byte, n int, charOffset bool) (int, int, error) { switch dir { case 0: lo = 0 hi = 0 fallthrough case '+': if charOffset { pos := hi for ; n > 0 && pos < len(data); n-- { _, size := utf8.DecodeRune(data[pos:]) pos += size } if n == 0 { return pos, pos, nil } break } // find next beginning of line if hi > 0 { for hi < len(data) && data[hi-1] != '\n' { hi++ } } lo = hi if n == 0 { return lo, hi, nil } for ; hi < len(data); hi++ { if data[hi] != '\n' { continue } switch n--; n { case 1: lo = hi + 1 case 0: return lo, hi + 1, nil } } case '-': if charOffset { // Scan backward for bytes that are not UTF-8 continuation bytes. pos := lo for ; pos > 0 && n > 0; pos-- { if data[pos]&0xc0 != 0x80 { n-- } } if n == 0 { return pos, pos, nil } break } // find earlier beginning of line for lo > 0 && data[lo-1] != '\n' { lo-- } hi = lo if n == 0 { return lo, hi, nil } for ; lo >= 0; lo-- { if lo > 0 && data[lo-1] != '\n' { continue } switch n--; n { case 1: hi = lo case 0: return lo, hi, nil } } } return 0, 0, errors.New("address out of range") } // addrRegexp searches for pattern in the given direction starting at lo, hi. // The direction dir is '+' (search forward from hi) or '-' (search backward from lo). // Backward searches are unimplemented. func addrRegexp(data []byte, lo, hi int, dir byte, pattern string) (int, int, error) { // We want ^ and $ to work as in sam/acme, so use ?m. re, err := regexp.Compile("(?m:" + pattern + ")") if err != nil { return 0, 0, err } if dir == '-' { // Could implement reverse search using binary search // through file, but that seems like overkill. return 0, 0, errors.New("reverse search not implemented") } m := re.FindIndex(data[hi:]) if len(m) > 0 { m[0] += hi m[1] += hi } else if hi > 0 { // No match. Wrap to beginning of data. m = re.FindIndex(data) } if len(m) == 0 { return 0, 0, errors.New("no match for " + pattern) } return m[0], m[1], nil } mmark-1.3-git849458a/doc_test.go000066400000000000000000000032201264073172300162510ustar00rootroot00000000000000// Parse document fragments. package mmark import "testing" var extensions = EXTENSION_TABLES | EXTENSION_FENCED_CODE | EXTENSION_AUTOLINK | EXTENSION_SPACE_HEADERS | EXTENSION_CITATION | EXTENSION_TITLEBLOCK_TOML | EXTENSION_HEADER_IDS | EXTENSION_AUTO_HEADER_IDS | EXTENSION_UNIQUE_HEADER_IDS | EXTENSION_FOOTNOTES | EXTENSION_SHORT_REF | EXTENSION_NO_EMPTY_LINE_BEFORE_BLOCK | EXTENSION_INCLUDE | EXTENSION_PARTS | EXTENSION_ABBREVIATIONS func TestCarriageReturn(t *testing.T) { var tests = []string{".# Abstract\n\rThis document\n\r# More\n\rand more\n\r{#fig-a}\n\r```\n\rfigure\n\r```\n\rFigure: Traditional Media Server\n\r\n\r{#fig-b}\n\r```\n\rfigure\n\r```\n\rFigure: Endpoint\n\r", "\n\n\nThis document\n\n\n\n\n
\nMore\n\nand more\n\n\n\nfigure\n\n\n\nFigure: Traditional Media Server\n\n\n\nfigure\n\n\n\nFigure: Endpoint\n\n
\n", } doTestsBlockXML(t, tests, extensions) } func TestFigureCaption(t *testing.T) { var tests = []string{ // This checks the *single* newline after the Figure: caption. ".# Abstract\nThis document\n# More\nand more\n{#fig-a}\n```\nfigure\n```\nFigure: Traditional Media Server\n\n{#fig-b}\n```\nfigure\n```\n", "\n\n\nThis document\n\n\n\n\n
\nMore\n\nand more\n\n\n
\nTraditional Media Server\n\n\nfigure\n\n
\n\nfigure\n\n
\n", } doTestsBlockXML(t, tests, extensions) } mmark-1.3-git849458a/html.go000066400000000000000000000751321264073172300154240ustar00rootroot00000000000000// HTML rendering backend package mmark import ( "bytes" xmllib "encoding/xml" "fmt" "io/ioutil" "sort" "strconv" "strings" ) // Html renderer configuration options. const ( HTML_SKIP_HTML = 1 << iota // skip preformatted HTML blocks HTML_SKIP_STYLE // skip embedded