summaryrefslogtreecommitdiffstats
path: root/runtime/doc/testing.txt
blob: 356d74e390b10f96780078f8a44d3071a2d6323a (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
*testing.txt*	For Vim version 9.1.  Last change: 2024 Feb 18


		  VIM REFERENCE MANUAL	  by Bram Moolenaar


Testing Vim and Vim script			*testing-support*

Expression evaluation is explained in |eval.txt|.  This file goes into details
about writing tests in Vim script.  This can be used for testing Vim itself
and for testing plugins.

1. Testing Vim				|testing|
2. Test functions			|test-functions-details|
3. Assert functions			|assert-functions-details|

==============================================================================
1. Testing Vim						*testing*

Vim can be tested after building it, usually with "make test".
The tests are located in the directory "src/testdir".

There are two types of tests added over time:
	test20.in		oldest, only for tiny builds
	test_something.vim	new style tests

						*new-style-testing*
New tests should be added as new style tests.  The test scripts are named
test_<feature>.vim (replace <feature> with the feature under test). These use
functions such as |assert_equal()| to keep the test commands and the expected
result in one place.
						*old-style-testing*
These tests are used only for testing Vim without the |+eval| feature.

Find more information in the file src/testdir/README.txt.

==============================================================================
2. Test functions				*test-functions-details*

test_alloc_fail({id}, {countdown}, {repeat})		*test_alloc_fail()*
		This is for testing: If the memory allocation with {id} is
		called, then decrement {countdown}, and when it reaches zero
		let memory allocation fail {repeat} times.  When {repeat} is
		smaller than one it fails one time.

		Can also be used as a |method|: >
			GetAllocId()->test_alloc_fail()


test_autochdir()					*test_autochdir()*
		Set a flag to enable the effect of 'autochdir' before Vim
		startup has finished.


test_feedinput({string})				*test_feedinput()*
		Characters in {string} are queued for processing as if they
		were typed by the user. This uses a low level input buffer.
		This function works only when with |+unix| or GUI is running.

		Can also be used as a |method|: >
			GetText()->test_feedinput()


test_garbagecollect_now()			 *test_garbagecollect_now()*
		Like garbagecollect(), but executed right away.  This must
		only be called directly to avoid any structure to exist
		internally, and |v:testing| must have been set before calling
		any function.   *E1142*
		This will not work when called from a :def function, because
		variables on the stack will be freed.


test_garbagecollect_soon()			 *test_garbagecollect_soon()*
		Set the flag to call the garbagecollector as if in the main
		loop.  Only to be used in tests.


test_getvalue({name})					*test_getvalue()*
		Get the value of an internal variable.  These values for
		{name} are supported:
			need_fileinfo

		Can also be used as a |method|: >
			GetName()->test_getvalue()
<
						*test_gui_event()*
test_gui_event({event}, {args})
		Generate a GUI {event} with arguments {args} for testing Vim
		functionality. This function works only when the GUI is
		running.

		{event} is a String and the supported values are:
		    "dropfiles"	drop one or more files in a window.
		    "findrepl"  search and replace text.
		    "mouse"	mouse button click event.
		    "scrollbar" move or drag the scrollbar.
		    "key"	send a low-level keyboard event.
		    "tabline"	select a tab page by mouse click.
		    "tabmenu"	select a tabline menu entry.

		{args} is a Dict and contains the arguments for the event.

		"dropfiles":
		  Drop one or more files in a specified window.  The supported
		  items in {args} are:
		    files:	List of file names
		    row:	window row number
		    col:	window column number
		    modifiers:	key modifiers. The supported values are:
				    0x4	Shift
				    0x8	Alt
				   0x10	Ctrl
		  The files are added to the |argument-list| and the first
		  file in {files} is edited in the window.  See |drag-n-drop|
		  for more information.  This event works only when the
		  |drop_file| feature is present.

		"findrepl":
		  {only available when the GUI has a find/replace dialog}
		  Perform a search and replace of text.  The supported items
		  in {args} are:
		    find_text:	string to find.
		    repl_text:	replacement string.
		    flags:	flags controlling the find/replace. Supported
				values are:
				    1	search next string (find dialog)
				    2	search next string (replace dialog)
				    3	replace string once
				    4	replace all matches
				    8	match whole words only
				   16	match case
		    forward:	set to 1 for forward search.

		"mouse":
		  Inject either a mouse button click, or a mouse move, event.
		  The supported items in {args} are:
		    button:	mouse button.  The supported values are:
				    0	left mouse button
				    1	middle mouse button
				    2	right mouse button
				    3	mouse button release
				    4	scroll wheel down
				    5	scroll wheel up
				    6	scroll wheel left
				    7	scroll wheel right
		    row:	mouse click row number.  The first row of the
				Vim window is 1 and the last row is 'lines'.
		    col:	mouse click column number.  The maximum value
				of {col} is 'columns'.
		    multiclick:	set to 1 to inject a multiclick mouse event.
		    modifiers:	key modifiers.  The supported values are:
				    4	shift is pressed
				    8	alt is pressed
				   16	ctrl is pressed
		    move:	Optional; if used and TRUE then a mouse move
			        event can be generated.
				Only {args} row: and col: are used and
				required; they are interpreted as pixels or
				screen cells, depending on "cell".
				Only results in an event when 'mousemoveevent'
				is set or a popup uses mouse move events.
		    cell:	Optional: when present and TRUE then "move"
				uses screen cells instead of pixel positions

		"scrollbar":
		  Set or drag the left, right or horizontal scrollbar.  Only
		  works when the scrollbar actually exists.  The supported
		  items in {args} are:
		    which:	Selects the scrollbar. The supported values
				are:
				    left  Left scrollbar of the current window
				    right Right scrollbar of the current window
				    hor   Horizontal scrollbar
		    value:	Amount to scroll.  For the vertical scrollbars
				the value can be between 0 to the line-count
				of the buffer minus one.  For the horizontal
				scrollbar the value can be between 1 and the
				maximum line length, assuming 'wrap' is not
				set.
		    dragging:	1 to drag the scrollbar and 0 to click in the
				scrollbar.

		"key":
		  Send a low-level keyboard event (e.g. key-up or down).
		  Currently only supported on MS-Windows.
		  The supported items in {args} are:
		    event:	The supported string values are:
				    keyup   generate a keyup event
				    keydown generate a keydown event
		    keycode:    Keycode to use for a keyup or a keydown event.
								*E1291*

		"tabline":
		  Inject a mouse click event on the tabline to select a
		  tabpage. The supported items in {args} are:
		    tabnr:	tab page number

		"tabmenu":
		  Inject an event to select a tabline menu entry. The
		  supported items in {args} are:
		    tabnr:	tab page number
		    item:	tab page menu item number. 1 for the first
				menu item, 2 for the second item and so on.

		After injecting the GUI events you probably should call
		|feedkeys()| to have them processed, e.g.: >
			call feedkeys("y", 'Lx!')
<
		Returns TRUE if the event is successfully added, FALSE if
		there is a failure.

		Can also be used as a |method|: >
			GetEvent()->test_gui_event({args})
<
test_ignore_error({expr})			 *test_ignore_error()*
		Ignore any error containing {expr}.  A normal message is given
		instead.
		This is only meant to be used in tests, where catching the
		error with try/catch cannot be used (because it skips over
		following code).
		{expr} is used literally, not as a pattern.
		When the {expr} is the string "RESET" then the list of ignored
		errors is made empty.

		Can also be used as a |method|: >
			GetErrorText()->test_ignore_error()


test_mswin_event({event}, {args})		*test_mswin_event()*
		Generate a low-level MS-Windows {event} with arguments {args}
		for testing Vim functionality.  It works for MS-Windows GUI
		and for the console.

		{event} is a String and the supported values are:
		    "mouse"	mouse event.
		    "key"	keyboard event.
		    "set_keycode_trans_strategy"
				Change the key translation method.

		"mouse":
		  Inject either a mouse button click, or a mouse move, event.
		  The supported items in {args} are:
		    button:	mouse button.  The supported values are:
				    0	right mouse button
				    1	middle mouse button
				    2	left mouse button
				    3	mouse button release
				    4	scroll wheel down
				    5	scroll wheel up
				    6	scroll wheel left
				    7	scroll wheel right
		    row:	mouse click row number.  The first row of the
				Vim window is 1 and the last row is 'lines'.
		    col:	mouse click column number.  The maximum value
				of {col} is 'columns'.
				Note: row and col are always interpreted as
				screen cells for the console application.
				But, they may be interpreted as pixels
				for the GUI, depending on "cell".
		    multiclick:	set to 1 to inject a double-click mouse event.
		    modifiers:	key modifiers.  The supported values are:
				    4	shift is pressed
				    8	alt is pressed
				   16	ctrl is pressed
		    move:	Optional; if used and TRUE then a mouse move
			        event can be generated.
				Only {args} row: and col: are used and
				required.
				Only results in an event when 'mousemoveevent'
				is set or a popup uses mouse move events.
		    cell:	Optional for the GUI: when present and TRUE
				then "move" uses screen cells instead of pixel
				positions.  Not used by the console.

		"key":
		  Send a low-level keyboard event (e.g. keyup or keydown).
		  The supported items in {args} are:
		    event:	The supported string values are:
				    keyup   generate a keyup event
				    keydown generate a keydown event
		    keycode:	Keycode to use for a keyup or a keydown event.
		    modifiers:	Optional; key modifiers.
				The supported values are:
				    2	shift is pressed
				    4	ctrl is pressed
				    8	alt is pressed
				Note: These values are different from the
				mouse modifiers.
		    execute:	Optional. Similar to |feedkeys()| mode x.
				When this is included and set to true
				(non-zero) then Vim will process any buffered
				unprocessed key events.  All other {args}
				items are optional when this is set and true.

		"set_keycode_trans_strategy":
		  |w32-experimental-keycode-trans-strategy|
		  Switch the keycode translation method. The supported methods
		  are:
		    experimental:   The method used after Patch v8.2.4807
				    using ToUnicode() Win API call.
		    classic:	    The method used pre Patch v8.2.4807
				    using the TranslateMessage() Win API call.

		Returns TRUE if the event is successfully added or executed,
		FALSE if there is a failure.

		Can also be used as a |method|: >
			GetEvent()->test_mswin_event({args})
<

test_null_blob()					*test_null_blob()*
		Return a |Blob| that is null. Only useful for testing.


test_null_channel()					*test_null_channel()*
		Return a |Channel| that is null. Only useful for testing.
		{only available when compiled with the +channel feature}


test_null_dict()					*test_null_dict()*
		Return a |Dict| that is null. Only useful for testing.


test_null_function()					*test_null_function()*
		Return a |Funcref| that is null. Only useful for testing.


test_null_job()						*test_null_job()*
		Return a |Job| that is null. Only useful for testing.
		{only available when compiled with the +job feature}


test_null_list()					*test_null_list()*
		Return a |List| that is null. Only useful for testing.


test_null_partial()					*test_null_partial()*
		Return a |Partial| that is null. Only useful for testing.


test_null_string()					*test_null_string()*
		Return a |String| that is null. Only useful for testing.


test_option_not_set({name})				*test_option_not_set()*
		Reset the flag that indicates option {name} was set.  Thus it
		looks like it still has the default value. Use like this: >
			set ambiwidth=double
			call test_option_not_set('ambiwidth')
<		Now the 'ambiwidth' option behaves like it was never changed,
		even though the value is "double".
		Only to be used for testing!

		Can also be used as a |method|: >
			GetOptionName()->test_option_not_set()


test_override({name}, {val})				*test_override()*
		Overrides certain parts of Vim's internal processing to be able
		to run tests. Only to be used for testing Vim!
		The override is enabled when {val} is non-zero and removed
		when {val} is zero.
		Current supported values for {name} are:

		{name}	     effect when {val} is non-zero ~
		alloc_lines  make a copy of every buffer line into allocated
			     memory, so that memory access errors can be found
			     by valgrind
		autoload     `import autoload` will load the script right
			     away, not postponed until an item is used
		char_avail   disable the char_avail() function
		nfa_fail     makes the NFA regexp engine fail to force a
			     fallback to the old engine
		no_query_mouse  do not query the mouse position for "dec"
				terminals
		no_wait_return	set the "no_wait_return" flag.  Not restored
				with "ALL".
		redraw       disable the redrawing() function
		redraw_flag  ignore the RedrawingDisabled flag
		starting     reset the "starting" variable, see below
		term_props   reset all terminal properties when the version
			     string is detected
		ui_delay     time in msec to use in ui_delay(); overrules a
			     wait time of up to 3 seconds for messages
		unreachable  no error for code after `:throw` and `:return`
		uptime	     overrules sysinfo.uptime
		vterm_title  setting the window title by a job running in a
			     terminal window
		ALL	     clear all overrides, except alloc_lines ({val} is
			     not used)

		"starting" is to be used when a test should behave like
		startup was done.  Since the tests are run by sourcing a
		script the "starting" variable is non-zero. This is usually a
		good thing (tests run faster), but sometimes this changes
		behavior in a way that the test doesn't work properly.
		When using: >
			call test_override('starting', 1)
<		The value of "starting" is saved.  It is restored by: >
			call test_override('starting', 0)

<		To make sure the flag is reset later using `:defer` can be
		useful: >
			call test_override('unreachable', 1)
			defer call test_override('unreachable', 0)

<		Can also be used as a |method|: >
			GetOverrideVal()-> test_override('starting')


test_refcount({expr})					*test_refcount()*
		Return the reference count of {expr}.  When {expr} is of a
		type that does not have a reference count, returns -1.  Only
		to be used for testing.

		Can also be used as a |method|: >
			GetVarname()->test_refcount()


test_setmouse({row}, {col})				*test_setmouse()*
		Set the mouse position to be used for the next mouse action.
		{row} and {col} are one based.
		For example: >
			call test_setmouse(4, 20)
			call feedkeys("\<LeftMouse>", "xt")


test_settime({expr})					*test_settime()*
		Set the time Vim uses internally.  Currently only used for
		timestamps in the history, as they are used in viminfo, and
		for undo.
		Using a value of 1 makes Vim not sleep after a warning or
		error message.
		{expr} must evaluate to a number.  When the value is zero the
		normal behavior is restored.

		Can also be used as a |method|: >
			GetTime()->test_settime()


test_srand_seed([seed])					*test_srand_seed()*
		When [seed] is given this sets the seed value used by
		`srand()`.  When omitted the test seed is removed.


test_unknown()						*test_unknown()*
		Return a value with unknown type. Only useful for testing.


test_void()						*test_void()*
		Return a value with void type. Only useful for testing.

==============================================================================
3. Assert functions				*assert-functions-details*


assert_beeps({cmd})					*assert_beeps()*
		Run {cmd} and add an error message to |v:errors| if it does
		NOT produce a beep or visual bell.
		Also see |assert_fails()|, |assert_nobeep()| and
		|assert-return|.

		Can also be used as a |method|: >
			GetCmd()->assert_beeps()
<
							*assert_equal()*
assert_equal({expected}, {actual} [, {msg}])
		When {expected} and {actual} are not equal an error message is
		added to |v:errors| and 1 is returned.  Otherwise zero is
		returned. |assert-return|
		The error is in the form "Expected {expected} but got
		{actual}".  When {msg} is present it is prefixed to that.

		There is no automatic conversion, the String "4" is different
		from the Number 4.  And the number 4 is different from the
		Float 4.0.  The value of 'ignorecase' is not used here, case
		always matters.
		Example: >
	assert_equal('foo', 'bar')
<		Will result in a string to be added to |v:errors|:
	test.vim line 12: Expected 'foo' but got 'bar' ~

		Can also be used as a |method|, the base is passed as the
		second argument: >
			mylist->assert_equal([1, 2, 3])

<							*assert_equalfile()*
assert_equalfile({fname-one}, {fname-two} [, {msg}])
		When the files {fname-one} and {fname-two} do not contain
		exactly the same text an error message is added to |v:errors|.
		Also see |assert-return|.
		When {fname-one} or {fname-two} does not exist the error will
		mention that.
		Mainly useful with |terminal-diff|.

		Can also be used as a |method|: >
			GetLog()->assert_equalfile('expected.log')

assert_exception({error} [, {msg}])			*assert_exception()*
		When v:exception does not contain the string {error} an error
		message is added to |v:errors|.  Also see |assert-return|.
		This can be used to assert that a command throws an exception.
		Using the error number, followed by a colon, avoids problems
		with translations: >
			try
			  commandthatfails
			  call assert_false(1, 'command should have failed')
			catch
			  call assert_exception('E492:')
			endtry
<
							*assert_fails()*
assert_fails({cmd} [, {error} [, {msg} [, {lnum} [, {context}]]]])
		Run {cmd} and add an error message to |v:errors| if it does
		NOT produce an error or when {error} is not found in the
		error message.  Also see |assert-return|.
							*E856*
		When {error} is a string it must be found literally in the
		first reported error. Most often this will be the error code,
		including the colon, e.g. "E123:". >
			assert_fails('bad cmd', 'E987:')
<
		When {error} is a |List| with one or two strings, these are
		used as patterns.  The first pattern is matched against the
		first reported error: >
			assert_fails('cmd', ['E987:.*expected bool'])
<		The second pattern, if present, is matched against the last
		reported error.
		If there is only one error then both patterns must match. This
		can be used to check that there is only one error.
		To only match the last error use an empty string for the first
		error: >
			assert_fails('cmd', ['', 'E987:'])
<
		If {msg} is empty then it is not used.  Do this to get the
		default message when passing the {lnum} argument.
							*E1115*
		When {lnum} is present and not negative, and the {error}
		argument is present and matches, then this is compared with
		the line number at which the error was reported. That can be
		the line number in a function or in a script.
							*E1116*
		When {context} is present it is used as a pattern and matched
		against the context (script name or function name) where
		{lnum} is located in.

		Note that beeping is not considered an error, and some failing
		commands only beep.  Use |assert_beeps()| for those.

		Can also be used as a |method|: >
			GetCmd()->assert_fails('E99:')

assert_false({actual} [, {msg}])			*assert_false()*
		When {actual} is not false an error message is added to
		|v:errors|, like with |assert_equal()|.
		The error is in the form "Expected False but got {actual}".
		When {msg} is present it is prepended to that.
		Also see |assert-return|.

		A value is false when it is zero. When {actual} is not a
		number the assert fails.

		Can also be used as a |method|: >
			GetResult()->assert_false()

assert_inrange({lower}, {upper}, {actual} [, {msg}])	 *assert_inrange()*
		This asserts number and |Float| values.  When {actual}  is lower
		than {lower} or higher than {upper} an error message is added
		to |v:errors|.  Also see |assert-return|.
		The error is in the form "Expected range {lower} - {upper},
		but got {actual}".  When {msg} is present it is prefixed to
		that.

								*assert_match()*
assert_match({pattern}, {actual} [, {msg}])
		When {pattern} does not match {actual} an error message is
		added to |v:errors|.  Also see |assert-return|.
		The error is in the form "Pattern {pattern} does not match
		{actual}".  When {msg} is present it is prefixed to that.

		{pattern} is used as with |=~|: The matching is always done
		like 'magic' was set and 'cpoptions' is empty, no matter what
		the actual value of 'magic' or 'cpoptions' is.

		{actual} is used as a string, automatic conversion applies.
		Use "^" and "$" to match with the start and end of the text.
		Use both to match the whole text.

		Example: >
	assert_match('^f.*o$', 'foobar')
<		Will result in a string to be added to |v:errors|:
	test.vim line 12: Pattern '^f.*o$' does not match 'foobar' ~

		Can also be used as a |method|: >
			getFile()->assert_match('foo.*')
<
assert_nobeep({cmd})					*assert_nobeep()*
		Run {cmd} and add an error message to |v:errors| if it
		produces a beep or visual bell.
		Also see |assert_beeps()|.

		Can also be used as a |method|: >
			GetCmd()->assert_nobeep()
<
							*assert_notequal()*
assert_notequal({expected}, {actual} [, {msg}])
		The opposite of `assert_equal()`: add an error message to
		|v:errors| when {expected} and {actual} are equal.
		Also see |assert-return|.

		Can also be used as a |method|: >
			mylist->assert_notequal([1, 2, 3])

<							*assert_notmatch()*
assert_notmatch({pattern}, {actual} [, {msg}])
		The opposite of `assert_match()`: add an error message to
		|v:errors| when {pattern} matches {actual}.
		Also see |assert-return|.

		Can also be used as a |method|: >
			getFile()->assert_notmatch('bar.*')


assert_report({msg})					*assert_report()*
		Report a test failure directly, using String {msg}.
		Always returns one.

		Can also be used as a |method|: >
			GetMessage()->assert_report()


assert_true({actual} [, {msg}])				*assert_true()*
		When {actual} is not true an error message is added to
		|v:errors|, like with |assert_equal()|.
		Also see |assert-return|.
		A value is TRUE when it is a non-zero number.  When {actual}
		is not a number the assert fails.
		When {msg} is given it precedes the default message.

		Can also be used as a |method|: >
			GetResult()->assert_true()
<

 vim:tw=78:ts=8:noet:ft=help:norl: