strange bug with packrat parsing

[mqnc]

It's been a while since I've last used peglib! The error reporting is amazing!

I want to write a program that parses python and I think I have encountered a bug when using packrat parsing.

Without packrat parsing everything works fine. With packrat parsing, after the code is parsed and LineBreak? CodeSegment from the first rule are matched, it tries to match another CodeSegment but for some reason from the middle of the input. My parser prints debug output to show this.

I am sorry I didn't reduce the grammar to narrow down the error but maybe you know what's going on right away and I can spare the effort.

Here is my parser:

#include "peglib.h"

#include <string>
#include <sstream>
#include <iostream>
#include <fstream>

using namespace peg;
using namespace std;

std::string readFile(std::string name){
	ifstream inputFile;
	inputFile.open(name);
	std::stringstream buffer("");
	buffer << inputFile.rdbuf();
	std::string content = buffer.str();
	inputFile.close();
	return content;
}

std::string filter(std::string s){
	std::stringstream result("");
	for(const auto &c:s){
		if('\t' == c){result << "→";}
		else if('\n' == c){result << "↲";}
		else if(' ' == c){result << "␣";}
		else{result << c;}
	}
	return result.str();
}

int main(void) {

	parser parser;

	parser.log = [](size_t line, size_t col, const string& msg) {
		cerr << line << ":" << col << ": " << msg << "\n";
	};

	std::string grammar = readFile("python.peg");
	auto ok = parser.load_grammar(grammar);

	if(!ok){
		cout << "could not create parser\n";
		return 1;
	}

	std::string source = readFile("test.py");

	//parser.enable_packrat_parsing();

	const auto rules = parser.get_rule_names();
	int indent = 0;
	for(auto& rule:rules){

		parser[rule.c_str()].enter = [rule, &indent](const char* s, size_t n, any& dt) {
			for(int i = 0; i < indent; i++){std::cout << "|  ";}
			std::cout << rule << " [?] " << filter(std::string(s)) << "\n";
			indent++;
		};

		parser[rule.c_str()] = [rule](const SemanticValues &vs) {
			if(vs.tokens.size() > 0){
				return vs.token_to_string();
			}
			else{
				auto result = std::stringstream("");
				for(const auto &v:vs){
					result << any_cast<std::string>(v);
				}
				return result.str();
			}
		};

		parser[rule.c_str()].leave = [rule, &indent](const char* s, size_t n, size_t matchlen, any& value, any& dt) {
			indent--;
			for(int i = 0; i < indent; i++){std::cout << "|  ";}
			std::cout << rule << (success(matchlen)? " [v] ":" [X] ") << filter(std::string(s)) << '\n';
		};
	}

	std::string output;
	std::cout << "parsing..." << "\n";
	ok = parser.parse(source, output);

	if(ok){
		cout << "parsing successful\n";
	}
	else{
		cout << "parsing not successful\n";
	}

	ofstream outputFile;
	outputFile.open("test.ff.py");
	outputFile << output;
	outputFile.close();

	return 0;
}

Here is my grammar python.peg:

Program <- $Indent<''> LineBreak? CodeSegment*
Wrong <- .*

Token <- (Misc / String / Group / !ColonBreak !LineBreak !Whitespace .)
Code <- Token (Whitespace / Token)*
ContinuingCode <- (Token / LineBreak)+
CodeLine <- <Code (LineBreak / EndOfFile)> # covers inline compound
Compound <- $(CompoundHeader MeasureMoreIndent CodeSegment IndentedCodeSegment*) CompoundEnd?
CompoundHeader <- <Code ColonBreak>
CompoundEnd <- SpaceTab* CompoundEndMarker

CodeSegment <- CodeLine / Compound
IndentedCodeSegment <- $Indent CodeSegment
MeasureMoreIndent <- $Indent<$Indent AdditionalIndent>
AdditionalIndent <-SpaceTab+

Group <- Parenthesized / Bracketed / Braced
Parenthesized <- '(' ContinuingCode? ')'
Bracketed <- '[' ContinuingCode? ']'
Braced <- '{' ContinuingCode? '}'

String <- TripleQuoteString / HexaQuoteString / SingleQuoteString / DoubleQuoteString
SingleQuoteString <- "'" (!"'" !'\n' StringCharacter)* "'"
DoubleQuoteString <- '"' (!'"' !'\n' StringCharacter)* '"'
TripleQuoteString <- "'''" (!"'''" StringCharacter)* "'''"
HexaQuoteString <- '"""' (!'"""' StringCharacter)* '"""'
StringCharacter <- ('\\' .) / .

SpaceTab <- ' ' / '\t'
Continuation <- '\\\n'
LineBreak <- (Whitespace? Comment? '\n')+
ColonBreak <- ':' LineBreak
Whitespace <- (SpaceTab / Continuation)+
Comment <- !CompoundEndMarker '#' (!'\n' .)*
CompoundEndMarker <- '#.\n'
EndOfFile <- !.

Misc <- [-!$%&*+,./0-9;<=>?@A-Z^_`a-z|~]+

here is my input test.py:

1:
	2:
		3
	4:
		5

and here is the debug output:

parsing...
Program [?] 1:↲→2:↲→→3↲→4:↲→→5↲
|  LineBreak [?] 1:↲→2:↲→→3↲→4:↲→→5↲
|  |  ...
|  LineBreak [X] 1:↲→2:↲→→3↲→4:↲→→5↲
|  CodeSegment [?] 1:↲→2:↲→→3↲→4:↲→→5↲
|  |  ...
|  CodeSegment [v] 1:↲→2:↲→→3↲→4:↲→→5↲
|  CodeSegment [?] →4:↲→→5↲
|  |  ...
|  CodeSegment [X] →4:↲→→5↲
Program [v] 1:↲→2:↲→→3↲→4:↲→→5↲
4:2: syntax error, unexpected '4'.
parsing not successful

[davidaucoin7377]

Review