Tokenization

This document provides technical specifications for the tokenization

Function overview

Functions for reading from input_ranges

read_until

read_line

read_one

read_raw_pod

read_number

read [???]

Functions for writing to output_ranges

write_until

write_line

write_one

write_raw_pod

write_number

write

Misc

split_by

template <typename input_t, typename delimiter_t, typename config_t = std::ignore>
    requires forward_range_concept<input_t> && predicate_concept<delimiter_t>
inline auto
split_by(input_t const & input,
         delimiter_t && delimiter,
         config_t && config)  // optional parameter
{
    /* implementation detail*/
    return // optional<view<view<sequence_type>>>
}

This function operates on a forward_range and returns view of views. The views can be empty if the sequence could not be split because the input might be empty. Otherwise the optional holds a view-of-views, so that no copying of sequence data is needed until the user explicitly assigns the return value to a proper container type to hold the data. This is also the reason, why input_range_concept is not applicable, as there is no guarantee that the seen data for tokenization is still present, when the iteration through the input continues.

crop_outer

crop_before_last

crop_before_first

find_last

find_first

wrap

skip_until

skip_line

skip_one

to_formatted_number