diff options
Diffstat (limited to 'Documentation/git-pack-objects.txt')
| -rw-r--r-- | Documentation/git-pack-objects.txt | 462 |
1 files changed, 462 insertions, 0 deletions
diff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt new file mode 100644 index 0000000..a9995a9 --- /dev/null +++ b/Documentation/git-pack-objects.txt @@ -0,0 +1,462 @@ +git-pack-objects(1) +=================== + +NAME +---- +git-pack-objects - Create a packed archive of objects + + +SYNOPSIS +-------- +[verse] +'git pack-objects' [-q | --progress | --all-progress] [--all-progress-implied] + [--no-reuse-delta] [--delta-base-offset] [--non-empty] + [--local] [--incremental] [--window=<n>] [--depth=<n>] + [--revs [--unpacked | --all]] [--keep-pack=<pack-name>] + [--cruft] [--cruft-expiration=<time>] + [--stdout [--filter=<filter-spec>] | <base-name>] + [--shallow] [--keep-true-parents] [--[no-]sparse] < <object-list> + + +DESCRIPTION +----------- +Reads list of objects from the standard input, and writes either one or +more packed archives with the specified base-name to disk, or a packed +archive to the standard output. + +A packed archive is an efficient way to transfer a set of objects +between two repositories as well as an access efficient archival +format. In a packed archive, an object is either stored as a +compressed whole or as a difference from some other object. +The latter is often called a delta. + +The packed archive format (.pack) is designed to be self-contained +so that it can be unpacked without any further information. Therefore, +each object that a delta depends upon must be present within the pack. + +A pack index file (.idx) is generated for fast, random access to the +objects in the pack. Placing both the index file (.idx) and the packed +archive (.pack) in the pack/ subdirectory of $GIT_OBJECT_DIRECTORY (or +any of the directories on $GIT_ALTERNATE_OBJECT_DIRECTORIES) +enables Git to read from the pack archive. + +The 'git unpack-objects' command can read the packed archive and +expand the objects contained in the pack into "one-file +one-object" format; this is typically done by the smart-pull +commands when a pack is created on-the-fly for efficient network +transport by their peers. + + +OPTIONS +------- +base-name:: + Write into pairs of files (.pack and .idx), using + <base-name> to determine the name of the created file. + When this option is used, the two files in a pair are written in + <base-name>-<SHA-1>.{pack,idx} files. <SHA-1> is a hash + based on the pack content and is written to the standard + output of the command. + +--stdout:: + Write the pack contents (what would have been written to + .pack file) out to the standard output. + +--revs:: + Read the revision arguments from the standard input, instead of + individual object names. The revision arguments are processed + the same way as 'git rev-list' with the `--objects` flag + uses its `commit` arguments to build the list of objects it + outputs. The objects on the resulting list are packed. + Besides revisions, `--not` or `--shallow <SHA-1>` lines are + also accepted. + +--unpacked:: + This implies `--revs`. When processing the list of + revision arguments read from the standard input, limit + the objects packed to those that are not already packed. + +--all:: + This implies `--revs`. In addition to the list of + revision arguments read from the standard input, pretend + as if all refs under `refs/` are specified to be + included. + +--include-tag:: + Include unasked-for annotated tags if the object they + reference was included in the resulting packfile. This + can be useful to send new tags to native Git clients. + +--stdin-packs:: + Read the basenames of packfiles (e.g., `pack-1234abcd.pack`) + from the standard input, instead of object names or revision + arguments. The resulting pack contains all objects listed in the + included packs (those not beginning with `^`), excluding any + objects listed in the excluded packs (beginning with `^`). ++ +Incompatible with `--revs`, or options that imply `--revs` (such as +`--all`), with the exception of `--unpacked`, which is compatible. + +--cruft:: + Packs unreachable objects into a separate "cruft" pack, denoted + by the existence of a `.mtimes` file. Typically used by `git + repack --cruft`. Callers provide a list of pack names and + indicate which packs will remain in the repository, along with + which packs will be deleted (indicated by the `-` prefix). The + contents of the cruft pack are all objects not contained in the + surviving packs which have not exceeded the grace period (see + `--cruft-expiration` below), or which have exceeded the grace + period, but are reachable from an other object which hasn't. ++ +When the input lists a pack containing all reachable objects (and lists +all other packs as pending deletion), the corresponding cruft pack will +contain all unreachable objects (with mtime newer than the +`--cruft-expiration`) along with any unreachable objects whose mtime is +older than the `--cruft-expiration`, but are reachable from an +unreachable object whose mtime is newer than the `--cruft-expiration`). ++ +Incompatible with `--unpack-unreachable`, `--keep-unreachable`, +`--pack-loose-unreachable`, `--stdin-packs`, as well as any other +options which imply `--revs`. Also incompatible with `--max-pack-size`; +when this option is set, the maximum pack size is not inferred from +`pack.packSizeLimit`. + +--cruft-expiration=<approxidate>:: + If specified, objects are eliminated from the cruft pack if they + have an mtime older than `<approxidate>`. If unspecified (and + given `--cruft`), then no objects are eliminated. + +--window=<n>:: +--depth=<n>:: + These two options affect how the objects contained in + the pack are stored using delta compression. The + objects are first internally sorted by type, size and + optionally names and compared against the other objects + within --window to see if using delta compression saves + space. --depth limits the maximum delta depth; making + it too deep affects the performance on the unpacker + side, because delta data needs to be applied that many + times to get to the necessary object. ++ +The default value for --window is 10 and --depth is 50. The maximum +depth is 4095. + +--window-memory=<n>:: + This option provides an additional limit on top of `--window`; + the window size will dynamically scale down so as to not take + up more than '<n>' bytes in memory. This is useful in + repositories with a mix of large and small objects to not run + out of memory with a large window, but still be able to take + advantage of the large window for the smaller objects. The + size can be suffixed with "k", "m", or "g". + `--window-memory=0` makes memory usage unlimited. The default + is taken from the `pack.windowMemory` configuration variable. + +--max-pack-size=<n>:: + In unusual scenarios, you may not be able to create files + larger than a certain size on your filesystem, and this option + can be used to tell the command to split the output packfile + into multiple independent packfiles, each not larger than the + given size. The size can be suffixed with + "k", "m", or "g". The minimum size allowed is limited to 1 MiB. + The default is unlimited, unless the config variable + `pack.packSizeLimit` is set. Note that this option may result in + a larger and slower repository; see the discussion in + `pack.packSizeLimit`. + +--honor-pack-keep:: + This flag causes an object already in a local pack that + has a .keep file to be ignored, even if it would have + otherwise been packed. + +--keep-pack=<pack-name>:: + This flag causes an object already in the given pack to be + ignored, even if it would have otherwise been + packed. `<pack-name>` is the pack file name without + leading directory (e.g. `pack-123.pack`). The option could be + specified multiple times to keep multiple packs. + +--incremental:: + This flag causes an object already in a pack to be ignored + even if it would have otherwise been packed. + +--local:: + This flag causes an object that is borrowed from an alternate + object store to be ignored even if it would have otherwise been + packed. + +--non-empty:: + Only create a packed archive if it would contain at + least one object. + +--progress:: + Progress status is reported on the standard error stream + by default when it is attached to a terminal, unless -q + is specified. This flag forces progress status even if + the standard error stream is not directed to a terminal. + +--all-progress:: + When --stdout is specified then progress report is + displayed during the object count and compression phases + but inhibited during the write-out phase. The reason is + that in some cases the output stream is directly linked + to another command which may wish to display progress + status of its own as it processes incoming pack data. + This flag is like --progress except that it forces progress + report for the write-out phase as well even if --stdout is + used. + +--all-progress-implied:: + This is used to imply --all-progress whenever progress display + is activated. Unlike --all-progress this flag doesn't actually + force any progress display by itself. + +-q:: + This flag makes the command not to report its progress + on the standard error stream. + +--no-reuse-delta:: + When creating a packed archive in a repository that + has existing packs, the command reuses existing deltas. + This sometimes results in a slightly suboptimal pack. + This flag tells the command not to reuse existing deltas + but compute them from scratch. + +--no-reuse-object:: + This flag tells the command not to reuse existing object data at all, + including non deltified object, forcing recompression of everything. + This implies --no-reuse-delta. Useful only in the obscure case where + wholesale enforcement of a different compression level on the + packed data is desired. + +--compression=<n>:: + Specifies compression level for newly-compressed data in the + generated pack. If not specified, pack compression level is + determined first by pack.compression, then by core.compression, + and defaults to -1, the zlib default, if neither is set. + Add --no-reuse-object if you want to force a uniform compression + level on all data no matter the source. + +--[no-]sparse:: + Toggle the "sparse" algorithm to determine which objects to include in + the pack, when combined with the "--revs" option. This algorithm + only walks trees that appear in paths that introduce new objects. + This can have significant performance benefits when computing + a pack to send a small change. However, it is possible that extra + objects are added to the pack-file if the included commits contain + certain types of direct renames. If this option is not included, + it defaults to the value of `pack.useSparse`, which is true unless + otherwise specified. + +--thin:: + Create a "thin" pack by omitting the common objects between a + sender and a receiver in order to reduce network transfer. This + option only makes sense in conjunction with --stdout. ++ +Note: A thin pack violates the packed archive format by omitting +required objects and is thus unusable by Git without making it +self-contained. Use `git index-pack --fix-thin` +(see linkgit:git-index-pack[1]) to restore the self-contained property. + +--shallow:: + Optimize a pack that will be provided to a client with a shallow + repository. This option, combined with --thin, can result in a + smaller pack at the cost of speed. + +--delta-base-offset:: + A packed archive can express the base object of a delta as + either a 20-byte object name or as an offset in the + stream, but ancient versions of Git don't understand the + latter. By default, 'git pack-objects' only uses the + former format for better compatibility. This option + allows the command to use the latter format for + compactness. Depending on the average delta chain + length, this option typically shrinks the resulting + packfile by 3-5 per-cent. ++ +Note: Porcelain commands such as `git gc` (see linkgit:git-gc[1]), +`git repack` (see linkgit:git-repack[1]) pass this option by default +in modern Git when they put objects in your repository into pack files. +So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle. + +--threads=<n>:: + Specifies the number of threads to spawn when searching for best + delta matches. This requires that pack-objects be compiled with + pthreads otherwise this option is ignored with a warning. + This is meant to reduce packing time on multiprocessor machines. + The required amount of memory for the delta search window is + however multiplied by the number of threads. + Specifying 0 will cause Git to auto-detect the number of CPU's + and set the number of threads accordingly. + +--index-version=<version>[,<offset>]:: + This is intended to be used by the test suite only. It allows + to force the version for the generated pack index, and to force + 64-bit index entries on objects located above the given offset. + +--keep-true-parents:: + With this option, parents that are hidden by grafts are packed + nevertheless. + +--filter=<filter-spec>:: + Requires `--stdout`. Omits certain objects (usually blobs) from + the resulting packfile. See linkgit:git-rev-list[1] for valid + `<filter-spec>` forms. + +--no-filter:: + Turns off any previous `--filter=` argument. + +--missing=<missing-action>:: + A debug option to help with future "partial clone" development. + This option specifies how missing objects are handled. ++ +The form '--missing=error' requests that pack-objects stop with an error if +a missing object is encountered. If the repository is a partial clone, an +attempt to fetch missing objects will be made before declaring them missing. +This is the default action. ++ +The form '--missing=allow-any' will allow object traversal to continue +if a missing object is encountered. No fetch of a missing object will occur. +Missing objects will silently be omitted from the results. ++ +The form '--missing=allow-promisor' is like 'allow-any', but will only +allow object traversal to continue for EXPECTED promisor missing objects. +No fetch of a missing object will occur. An unexpected missing object will +raise an error. + +--exclude-promisor-objects:: + Omit objects that are known to be in the promisor remote. (This + option has the purpose of operating only on locally created objects, + so that when we repack, we still maintain a distinction between + locally created objects [without .promisor] and objects from the + promisor remote [with .promisor].) This is used with partial clone. + +--keep-unreachable:: + Objects unreachable from the refs in packs named with + --unpacked= option are added to the resulting pack, in + addition to the reachable objects that are not in packs marked + with *.keep files. This implies `--revs`. + +--pack-loose-unreachable:: + Pack unreachable loose objects (and their loose counterparts + removed). This implies `--revs`. + +--unpack-unreachable:: + Keep unreachable objects in loose form. This implies `--revs`. + +--delta-islands:: + Restrict delta matches based on "islands". See DELTA ISLANDS + below. + + +DELTA ISLANDS +------------- + +When possible, `pack-objects` tries to reuse existing on-disk deltas to +avoid having to search for new ones on the fly. This is an important +optimization for serving fetches, because it means the server can avoid +inflating most objects at all and just send the bytes directly from +disk. This optimization can't work when an object is stored as a delta +against a base which the receiver does not have (and which we are not +already sending). In that case the server "breaks" the delta and has to +find a new one, which has a high CPU cost. Therefore it's important for +performance that the set of objects in on-disk delta relationships match +what a client would fetch. + +In a normal repository, this tends to work automatically. The objects +are mostly reachable from the branches and tags, and that's what clients +fetch. Any deltas we find on the server are likely to be between objects +the client has or will have. + +But in some repository setups, you may have several related but separate +groups of ref tips, with clients tending to fetch those groups +independently. For example, imagine that you are hosting several "forks" +of a repository in a single shared object store, and letting clients +view them as separate repositories through `GIT_NAMESPACE` or separate +repos using the alternates mechanism. A naive repack may find that the +optimal delta for an object is against a base that is only found in +another fork. But when a client fetches, they will not have the base +object, and we'll have to find a new delta on the fly. + +A similar situation may exist if you have many refs outside of +`refs/heads/` and `refs/tags/` that point to related objects (e.g., +`refs/pull` or `refs/changes` used by some hosting providers). By +default, clients fetch only heads and tags, and deltas against objects +found only in those other groups cannot be sent as-is. + +Delta islands solve this problem by allowing you to group your refs into +distinct "islands". Pack-objects computes which objects are reachable +from which islands, and refuses to make a delta from an object `A` +against a base which is not present in all of `A`'s islands. This +results in slightly larger packs (because we miss some delta +opportunities), but guarantees that a fetch of one island will not have +to recompute deltas on the fly due to crossing island boundaries. + +When repacking with delta islands the delta window tends to get +clogged with candidates that are forbidden by the config. Repacking +with a big --window helps (and doesn't take as long as it otherwise +might because we can reject some object pairs based on islands before +doing any computation on the content). + +Islands are configured via the `pack.island` option, which can be +specified multiple times. Each value is a left-anchored regular +expressions matching refnames. For example: + +------------------------------------------- +[pack] +island = refs/heads/ +island = refs/tags/ +------------------------------------------- + +puts heads and tags into an island (whose name is the empty string; see +below for more on naming). Any refs which do not match those regular +expressions (e.g., `refs/pull/123`) is not in any island. Any object +which is reachable only from `refs/pull/` (but not heads or tags) is +therefore not a candidate to be used as a base for `refs/heads/`. + +Refs are grouped into islands based on their "names", and two regexes +that produce the same name are considered to be in the same +island. The names are computed from the regexes by concatenating any +capture groups from the regex, with a '-' dash in between. (And if +there are no capture groups, then the name is the empty string, as in +the above example.) This allows you to create arbitrary numbers of +islands. Only up to 14 such capture groups are supported though. + +For example, imagine you store the refs for each fork in +`refs/virtual/ID`, where `ID` is a numeric identifier. You might then +configure: + +------------------------------------------- +[pack] +island = refs/virtual/([0-9]+)/heads/ +island = refs/virtual/([0-9]+)/tags/ +island = refs/virtual/([0-9]+)/(pull)/ +------------------------------------------- + +That puts the heads and tags for each fork in their own island (named +"1234" or similar), and the pull refs for each go into their own +"1234-pull". + +Note that we pick a single island for each regex to go into, using "last +one wins" ordering (which allows repo-specific config to take precedence +over user-wide config, and so forth). + + +CONFIGURATION +------------- + +Various configuration variables affect packing, see +linkgit:git-config[1] (search for "pack" and "delta"). + +Notably, delta compression is not used on objects larger than the +`core.bigFileThreshold` configuration variable and on files with the +attribute `delta` set to false. + +SEE ALSO +-------- +linkgit:git-rev-list[1] +linkgit:git-repack[1] +linkgit:git-prune-packed[1] + +GIT +--- +Part of the linkgit:git[1] suite |