1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
|
``segmenter_word::ffi``
=======================
.. cpp:enum-struct:: ICU4XSegmenterWordType
See the `Rust documentation for WordType <https://docs.rs/icu/latest/icu/segmenter/enum.WordType.html>`__ for more information.
.. cpp:enumerator:: None
.. cpp:enumerator:: Number
.. cpp:enumerator:: Letter
.. cpp:class:: ICU4XWordBreakIteratorLatin1
See the `Rust documentation for WordBreakIterator <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html>`__ for more information.
.. cpp:function:: int32_t next()
Finds the next breakpoint. Returns -1 if at the end of the string or if the index is out of range of a 32-bit signed integer.
See the `Rust documentation for next <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html#method.next>`__ for more information.
.. cpp:function:: ICU4XSegmenterWordType word_type() const
Return the status value of break boundary.
See the `Rust documentation for word_type <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html#method.word_type>`__ for more information.
.. cpp:function:: bool is_word_like() const
Return true when break boundary is word-like such as letter/number/CJK
See the `Rust documentation for is_word_like <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html#method.is_word_like>`__ for more information.
.. cpp:class:: ICU4XWordBreakIteratorUtf16
See the `Rust documentation for WordBreakIterator <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html>`__ for more information.
.. cpp:function:: int32_t next()
Finds the next breakpoint. Returns -1 if at the end of the string or if the index is out of range of a 32-bit signed integer.
See the `Rust documentation for next <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html#method.next>`__ for more information.
.. cpp:function:: ICU4XSegmenterWordType word_type() const
Return the status value of break boundary.
See the `Rust documentation for word_type <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html#method.word_type>`__ for more information.
.. cpp:function:: bool is_word_like() const
Return true when break boundary is word-like such as letter/number/CJK
See the `Rust documentation for is_word_like <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html#method.is_word_like>`__ for more information.
.. cpp:class:: ICU4XWordBreakIteratorUtf8
See the `Rust documentation for WordBreakIterator <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html>`__ for more information.
.. cpp:function:: int32_t next()
Finds the next breakpoint. Returns -1 if at the end of the string or if the index is out of range of a 32-bit signed integer.
See the `Rust documentation for next <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html#method.next>`__ for more information.
.. cpp:function:: ICU4XSegmenterWordType word_type() const
Return the status value of break boundary.
See the `Rust documentation for word_type <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html#method.word_type>`__ for more information.
.. cpp:function:: bool is_word_like() const
Return true when break boundary is word-like such as letter/number/CJK
See the `Rust documentation for is_word_like <https://docs.rs/icu/latest/icu/segmenter/struct.WordBreakIterator.html#method.is_word_like>`__ for more information.
.. cpp:class:: ICU4XWordSegmenter
An ICU4X word-break segmenter, capable of finding word breakpoints in strings.
See the `Rust documentation for WordSegmenter <https://docs.rs/icu/latest/icu/segmenter/struct.WordSegmenter.html>`__ for more information.
.. cpp:function:: static diplomat::result<ICU4XWordSegmenter, ICU4XError> create_auto(const ICU4XDataProvider& provider)
Construct an :cpp:class:`ICU4XWordSegmenter` with automatically selecting the best available LSTM or dictionary payload data.
Note: currently, it uses dictionary for Chinese and Japanese, and LSTM for Burmese, Khmer, Lao, and Thai.
See the `Rust documentation for new_auto <https://docs.rs/icu/latest/icu/segmenter/struct.WordSegmenter.html#method.new_auto>`__ for more information.
.. cpp:function:: static diplomat::result<ICU4XWordSegmenter, ICU4XError> create_lstm(const ICU4XDataProvider& provider)
Construct an :cpp:class:`ICU4XWordSegmenter` with LSTM payload data for Burmese, Khmer, Lao, and Thai.
Warning: :cpp:class:`ICU4XWordSegmenter` created by this function doesn't handle Chinese or Japanese.
See the `Rust documentation for new_lstm <https://docs.rs/icu/latest/icu/segmenter/struct.WordSegmenter.html#method.new_lstm>`__ for more information.
.. cpp:function:: static diplomat::result<ICU4XWordSegmenter, ICU4XError> create_dictionary(const ICU4XDataProvider& provider)
Construct an :cpp:class:`ICU4XWordSegmenter` with dictionary payload data for Chinese, Japanese, Burmese, Khmer, Lao, and Thai.
See the `Rust documentation for new_dictionary <https://docs.rs/icu/latest/icu/segmenter/struct.WordSegmenter.html#method.new_dictionary>`__ for more information.
.. cpp:function:: ICU4XWordBreakIteratorUtf8 segment_utf8(const std::string_view input) const
Segments a (potentially ill-formed) UTF-8 string.
See the `Rust documentation for segment_utf8 <https://docs.rs/icu/latest/icu/segmenter/struct.WordSegmenter.html#method.segment_utf8>`__ for more information.
Lifetimes: ``this``, ``input`` must live at least as long as the output.
.. cpp:function:: ICU4XWordBreakIteratorUtf16 segment_utf16(const diplomat::span<const uint16_t> input) const
Segments a UTF-16 string.
See the `Rust documentation for segment_utf16 <https://docs.rs/icu/latest/icu/segmenter/struct.WordSegmenter.html#method.segment_utf16>`__ for more information.
Lifetimes: ``this``, ``input`` must live at least as long as the output.
.. cpp:function:: ICU4XWordBreakIteratorLatin1 segment_latin1(const diplomat::span<const uint8_t> input) const
Segments a Latin-1 string.
See the `Rust documentation for segment_latin1 <https://docs.rs/icu/latest/icu/segmenter/struct.WordSegmenter.html#method.segment_latin1>`__ for more information.
Lifetimes: ``this``, ``input`` must live at least as long as the output.
|
