Compilers Part 5, working with in memory data buffers

In the previous entries, we were able to set up the spec for the scripting language. To port the interpreter into a tcp/ip server, the first task is to allow the lexer to work with in memory data buffers instead of stdin and stdout. The reason is simple, user input will come from a network client and there are subtle differences between a network socket and stdin.

Fortunately, flex provides several interface to set up in memory data buffer as token input. The following lex source code demonstrates how to use the relevant interface:

%{
extern "C"{
#include < sys/stat.h>
#include < fcntl.h>
#include < string.h>
}
#include < iostream>
#include < sstream>
#include < fstream>
#include < string>
#include < vector>
#include < algorithm>
using namespace std;

unsigned int line = 0;
std::vector< std::string> text;

%}

extern int yywrap();
%%

\/\/.*$    { cout << "comment: " << yytext; }
.|\n        ;
%%

YY_BUFFER_STATE cur_buffer;
int main(int argc, char * argv[]){

    cout << argv[1] << '\n';
    ifstream ifs(argv[1]);

    char buf[256];
    int len;
    while(ifs.good()){
        memset(buf, 0, 256);
        ifs.getline(buf, 254);
        len = strlen(buf);
        buf[len] = '\n';
        text.push_back(buf);
        cout << buf;
    }
    cout << "\nlines read: " << text.size() << '\n';

    cur_buffer = yy_scan_string(text[line].c_str());
    extern int yylex();
    yylex();

    return 0;
}

int yywrap(){

    yy_delete_buffer(cur_buffer);
    if(line+1 > text.size()) return 1;
    cur_buffer = yy_scan_string(text[line].c_str());
    line ++;
    return 0;
}

yywrap gets called by yylex whenever a input buffer is exhausted, if yywrap returns 1, yylex will return; Therefore, it's a common technique to set up another available data buffer and return 0 to allow yylex continue processing as done in this example.

References:
1. http://flex.sourceforge.net/manual/Multiple-Input-Buffers.html#Multiple-Input-Buffers