diff options
| author | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-05-04 12:17:33 +0000 |
|---|---|---|
| committer | Daniel Baumann <daniel.baumann@progress-linux.org> | 2024-05-04 12:17:33 +0000 |
| commit | 5e45211a64149b3c659b90ff2de6fa982a5a93ed (patch) | |
| tree | 739caf8c461053357daa9f162bef34516c7bf452 /doc/src/sgml/html/logicaldecoding-streaming.html | |
| parent | Initial commit. (diff) | |
| download | postgresql-15-5e45211a64149b3c659b90ff2de6fa982a5a93ed.tar.xz postgresql-15-5e45211a64149b3c659b90ff2de6fa982a5a93ed.zip | |
Adding upstream version 15.5.upstream/15.5
Signed-off-by: Daniel Baumann <daniel.baumann@progress-linux.org>
Diffstat (limited to 'doc/src/sgml/html/logicaldecoding-streaming.html')
| -rw-r--r-- | doc/src/sgml/html/logicaldecoding-streaming.html | 86 |
1 files changed, 86 insertions, 0 deletions
diff --git a/doc/src/sgml/html/logicaldecoding-streaming.html b/doc/src/sgml/html/logicaldecoding-streaming.html new file mode 100644 index 0000000..c1ca812 --- /dev/null +++ b/doc/src/sgml/html/logicaldecoding-streaming.html @@ -0,0 +1,86 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"><html xmlns="http://www.w3.org/1999/xhtml"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /><title>49.9. Streaming of Large Transactions for Logical Decoding</title><link rel="stylesheet" type="text/css" href="stylesheet.css" /><link rev="made" href="pgsql-docs@lists.postgresql.org" /><meta name="generator" content="DocBook XSL Stylesheets Vsnapshot" /><link rel="prev" href="logicaldecoding-synchronous.html" title="49.8. Synchronous Replication Support for Logical Decoding" /><link rel="next" href="logicaldecoding-two-phase-commits.html" title="49.10. Two-phase Commit Support for Logical Decoding" /></head><body id="docContent" class="container-fluid col-10"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="5" align="center">49.9. Streaming of Large Transactions for Logical Decoding</th></tr><tr><td width="10%" align="left"><a accesskey="p" href="logicaldecoding-synchronous.html" title="49.8. Synchronous Replication Support for Logical Decoding">Prev</a> </td><td width="10%" align="left"><a accesskey="u" href="logicaldecoding.html" title="Chapter 49. Logical Decoding">Up</a></td><th width="60%" align="center">Chapter 49. Logical Decoding</th><td width="10%" align="right"><a accesskey="h" href="index.html" title="PostgreSQL 15.5 Documentation">Home</a></td><td width="10%" align="right"> <a accesskey="n" href="logicaldecoding-two-phase-commits.html" title="49.10. Two-phase Commit Support for Logical Decoding">Next</a></td></tr></table><hr /></div><div class="sect1" id="LOGICALDECODING-STREAMING"><div class="titlepage"><div><div><h2 class="title" style="clear: both">49.9. Streaming of Large Transactions for Logical Decoding</h2></div></div></div><p> + The basic output plugin callbacks (e.g., <code class="function">begin_cb</code>, + <code class="function">change_cb</code>, <code class="function">commit_cb</code> and + <code class="function">message_cb</code>) are only invoked when the transaction + actually commits. The changes are still decoded from the transaction + log, but are only passed to the output plugin at commit (and discarded + if the transaction aborts). + </p><p> + This means that while the decoding happens incrementally, and may spill + to disk to keep memory usage under control, all the decoded changes have + to be transmitted when the transaction finally commits (or more precisely, + when the commit is decoded from the transaction log). Depending on the + size of the transaction and network bandwidth, the transfer time may + significantly increase the apply lag. + </p><p> + To reduce the apply lag caused by large transactions, an output plugin + may provide additional callback to support incremental streaming of + in-progress transactions. There are multiple required streaming callbacks + (<code class="function">stream_start_cb</code>, <code class="function">stream_stop_cb</code>, + <code class="function">stream_abort_cb</code>, <code class="function">stream_commit_cb</code> + and <code class="function">stream_change_cb</code>) and two optional callbacks + (<code class="function">stream_message_cb</code> and <code class="function">stream_truncate_cb</code>). + Also, if streaming of two-phase commands is to be supported, then additional + callbacks must be provided. (See <a class="xref" href="logicaldecoding-two-phase-commits.html" title="49.10. Two-phase Commit Support for Logical Decoding">Section 49.10</a> + for details). + </p><p> + When streaming an in-progress transaction, the changes (and messages) are + streamed in blocks demarcated by <code class="function">stream_start_cb</code> + and <code class="function">stream_stop_cb</code> callbacks. Once all the decoded + changes are transmitted, the transaction can be committed using the + <code class="function">stream_commit_cb</code> callback + (or possibly aborted using the <code class="function">stream_abort_cb</code> callback). + If two-phase commits are supported, the transaction can be prepared using the + <code class="function">stream_prepare_cb</code> callback, + <code class="command">COMMIT PREPARED</code> using the + <code class="function">commit_prepared_cb</code> callback or aborted using the + <code class="function">rollback_prepared_cb</code>. + </p><p> + One example sequence of streaming callback calls for one transaction may + look like this: +</p><pre class="programlisting"> +stream_start_cb(...); <-- start of first block of changes + stream_change_cb(...); + stream_change_cb(...); + stream_message_cb(...); + stream_change_cb(...); + ... + stream_change_cb(...); +stream_stop_cb(...); <-- end of first block of changes + +stream_start_cb(...); <-- start of second block of changes + stream_change_cb(...); + stream_change_cb(...); + stream_change_cb(...); + ... + stream_message_cb(...); + stream_change_cb(...); +stream_stop_cb(...); <-- end of second block of changes + + +[a. when using normal commit] +stream_commit_cb(...); <-- commit of the streamed transaction + +[b. when using two-phase commit] +stream_prepare_cb(...); <-- prepare the streamed transaction +commit_prepared_cb(...); <-- commit of the prepared transaction +</pre><p> + </p><p> + The actual sequence of callback calls may be more complicated, of course. + There may be blocks for multiple streamed transactions, some of the + transactions may get aborted, etc. + </p><p> + Similar to spill-to-disk behavior, streaming is triggered when the total + amount of changes decoded from the WAL (for all in-progress transactions) + exceeds the limit defined by <code class="varname">logical_decoding_work_mem</code> setting. + At that point, the largest top-level transaction (measured by the amount of memory + currently used for decoded changes) is selected and streamed. However, in + some cases we still have to spill to disk even if streaming is enabled + because we exceed the memory threshold but still have not decoded the + complete tuple e.g., only decoded toast table insert but not the main table + insert. + </p><p> + Even when streaming large transactions, the changes are still applied in + commit order, preserving the same guarantees as the non-streaming mode. + </p></div><div class="navfooter"><hr /><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="logicaldecoding-synchronous.html" title="49.8. Synchronous Replication Support for Logical Decoding">Prev</a> </td><td width="20%" align="center"><a accesskey="u" href="logicaldecoding.html" title="Chapter 49. Logical Decoding">Up</a></td><td width="40%" align="right"> <a accesskey="n" href="logicaldecoding-two-phase-commits.html" title="49.10. Two-phase Commit Support for Logical Decoding">Next</a></td></tr><tr><td width="40%" align="left" valign="top">49.8. Synchronous Replication Support for Logical Decoding </td><td width="20%" align="center"><a accesskey="h" href="index.html" title="PostgreSQL 15.5 Documentation">Home</a></td><td width="40%" align="right" valign="top"> 49.10. Two-phase Commit Support for Logical Decoding</td></tr></table></div></body></html>
\ No newline at end of file |