| Safe Haskell | Safe |
|---|---|
| Language | Haskell2010 |
Hpp.Tokens
Description
Tokenization breaks a String into pieces of whitespace,
constants, symbols, and identifiers.
- data Token
- detok :: Token -> String
- isImportant :: Token -> Bool
- notImportant :: Token -> Bool
- importants :: [Token] -> [String]
- trimUnimportant :: [Token] -> [Token]
- newLine :: Token -> Bool
- tokWords :: String -> [Token]
- skipLiteral :: ((String -> String) -> String -> r) -> String -> r
- splits :: (Char -> Bool) -> String -> [String]
- validIdentifierChar :: Char -> Bool
- fixExponents :: [Token] -> [Token]
- tokenize :: String -> [Token]
- detokenize :: [Token] -> String
Documentation
Tokenization is words except the white space is tagged rather
than discarded.
skipLiteral :: ((String -> String) -> String -> r) -> String -> r Source #
If you encounter a string literal, call this helper with a double-barreled continuation and the rest of your input. The continuation will expect the remainder of the string literal as the first argument, and the remaining input as the second argument.
splits :: (Char -> Bool) -> String -> [String] Source #
splits isDelimiter str tokenizes str using isDelimiter as a
delimiter predicate. Leading whitespace is also stripped from
tokens.
validIdentifierChar :: Char -> Bool Source #
Predicate on space characters based on something approximating valid identifier syntax. This is used to break apart non-space characters.
fixExponents :: [Token] -> [Token] Source #
Something like 12E+FOO is a single pre-processor token, so
FOO should not be macro expanded.