diff options
Diffstat (limited to '')
33 files changed, 6637 insertions, 0 deletions
diff --git a/Documentation/block/00-INDEX b/Documentation/block/00-INDEX new file mode 100644 index 000000000..8d55b4bbb --- /dev/null +++ b/Documentation/block/00-INDEX @@ -0,0 +1,34 @@ +00-INDEX + - This file +bfq-iosched.txt + - BFQ IO scheduler and its tunables +biodoc.txt + - Notes on the Generic Block Layer Rewrite in Linux 2.5 +biovecs.txt + - Immutable biovecs and biovec iterators +capability.txt + - Generic Block Device Capability (/sys/block/<device>/capability) +cfq-iosched.txt + - CFQ IO scheduler tunables +cmdline-partition.txt + - how to specify block device partitions on kernel command line +data-integrity.txt + - Block data integrity +deadline-iosched.txt + - Deadline IO scheduler tunables +ioprio.txt + - Block io priorities (in CFQ scheduler) +pr.txt + - Block layer support for Persistent Reservations +null_blk.txt + - Null block for block-layer benchmarking. +queue-sysfs.txt + - Queue's sysfs entries +request.txt + - The members of struct request (in include/linux/blkdev.h) +stat.txt + - Block layer statistics in /sys/block/<device>/stat +switching-sched.txt + - Switching I/O schedulers at runtime +writeback_cache_control.txt + - Control of volatile write back caches diff --git a/Documentation/block/bfq-iosched.txt b/Documentation/block/bfq-iosched.txt new file mode 100644 index 000000000..8d8d8f06c --- /dev/null +++ b/Documentation/block/bfq-iosched.txt @@ -0,0 +1,561 @@ +BFQ (Budget Fair Queueing) +========================== + +BFQ is a proportional-share I/O scheduler, with some extra +low-latency capabilities. In addition to cgroups support (blkio or io +controllers), BFQ's main features are: +- BFQ guarantees a high system and application responsiveness, and a + low latency for time-sensitive applications, such as audio or video + players; +- BFQ distributes bandwidth, and not just time, among processes or + groups (switching back to time distribution when needed to keep + throughput high). + +In its default configuration, BFQ privileges latency over +throughput. So, when needed for achieving a lower latency, BFQ builds +schedules that may lead to a lower throughput. If your main or only +goal, for a given device, is to achieve the maximum-possible +throughput at all times, then do switch off all low-latency heuristics +for that device, by setting low_latency to 0. See Section 3 for +details on how to configure BFQ for the desired tradeoff between +latency and throughput, or on how to maximize throughput. + +BFQ has a non-null overhead, which limits the maximum IOPS that a CPU +can process for a device scheduled with BFQ. To give an idea of the +limits on slow or average CPUs, here are, first, the limits of BFQ for +three different CPUs, on, respectively, an average laptop, an old +desktop, and a cheap embedded system, in case full hierarchical +support is enabled (i.e., CONFIG_BFQ_GROUP_IOSCHED is set), but +CONFIG_DEBUG_BLK_CGROUP is not set (Section 4-2): +- Intel i7-4850HQ: 400 KIOPS +- AMD A8-3850: 250 KIOPS +- ARM CortexTM-A53 Octa-core: 80 KIOPS + +If CONFIG_DEBUG_BLK_CGROUP is set (and of course full hierarchical +support is enabled), then the sustainable throughput with BFQ +decreases, because all blkio.bfq* statistics are created and updated +(Section 4-2). For BFQ, this leads to the following maximum +sustainable throughputs, on the same systems as above: +- Intel i7-4850HQ: 310 KIOPS +- AMD A8-3850: 200 KIOPS +- ARM CortexTM-A53 Octa-core: 56 KIOPS + +BFQ works for multi-queue devices too. + +The table of contents follow. Impatients can just jump to Section 3. + +CONTENTS + +1. When may BFQ be useful? + 1-1 Personal systems + 1-2 Server systems +2. How does BFQ work? +3. What are BFQ's tunables and how to properly configure BFQ? +4. BFQ group scheduling + 4-1 Service guarantees provided + 4-2 Interface + +1. When may BFQ be useful? +========================== + +BFQ provides the following benefits on personal and server systems. + +1-1 Personal systems +-------------------- + +Low latency for interactive applications + +Regardless of the actual background workload, BFQ guarantees that, for +interactive tasks, the storage device is virtually as responsive as if +it was idle. For example, even if one or more of the following +background workloads are being executed: +- one or more large files are being read, written or copied, +- a tree of source files is being compiled, +- one or more virtual machines are performing I/O, +- a software update is in progress, +- indexing daemons are scanning filesystems and updating their + databases, +starting an application or loading a file from within an application +takes about the same time as if the storage device was idle. As a +comparison, with CFQ, NOOP or DEADLINE, and in the same conditions, +applications experience high latencies, or even become unresponsive +until the background workload terminates (also on SSDs). + +Low latency for soft real-time applications + +Also soft real-time applications, such as audio and video +players/streamers, enjoy a low latency and a low drop rate, regardless +of the background I/O workload. As a consequence, these applications +do not suffer from almost any glitch due to the background workload. + +Higher speed for code-development tasks + +If some additional workload happens to be executed in parallel, then +BFQ executes the I/O-related components of typical code-development +tasks (compilation, checkout, merge, ...) much more quickly than CFQ, +NOOP or DEADLINE. + +High throughput + +On hard disks, BFQ achieves up to 30% higher throughput than CFQ, and +up to 150% higher throughput than DEADLINE and NOOP, with all the +sequential workloads considered in our tests. With random workloads, +and with all the workloads on flash-based devices, BFQ achieves, +instead, about the same throughput as the other schedulers. + +Strong fairness, bandwidth and delay guarantees + +BFQ distributes the device throughput, and not just the device time, +among I/O-bound applications in proportion their weights, with any +workload and regardless of the device parameters. From these bandwidth +guarantees, it is possible to compute tight per-I/O-request delay +guarantees by a simple formula. If not configured for strict service +guarantees, BFQ switches to time-based resource sharing (only) for +applications that would otherwise cause a throughput loss. + +1-2 Server systems +------------------ + +Most benefits for server systems follow from the same service +properties as above. In particular, regardless of whether additional, +possibly heavy workloads are being served, BFQ guarantees: + +. audio and video-streaming with zero or very low jitter and drop + rate; + +. fast retrieval of WEB pages and embedded objects; + +. real-time recording of data in live-dumping applications (e.g., + packet logging); + +. responsiveness in local and remote access to a server. + + +2. How does BFQ work? +===================== + +BFQ is a proportional-share I/O scheduler, whose general structure, +plus a lot of code, are borrowed from CFQ. + +- Each process doing I/O on a device is associated with a weight and a + (bfq_)queue. + +- BFQ grants exclusive access to the device, for a while, to one queue + (process) at a time, and implements this service model by + associating every queue with a budget, measured in number of + sectors. + + - After a queue is granted access to the device, the budget of the + queue is decremented, on each request dispatch, by the size of the + request. + + - The in-service queue is expired, i.e., its service is suspended, + only if one of the following events occurs: 1) the queue finishes + its budget, 2) the queue empties, 3) a "budget timeout" fires. + + - The budget timeout prevents processes doing random I/O from + holding the device for too long and dramatically reducing + throughput. + + - Actually, as in CFQ, a queue associated with a process issuing + sync requests may not be expired immediately when it empties. In + contrast, BFQ may idle the device for a short time interval, + giving the process the chance to go on being served if it issues + a new request in time. Device idling typically boosts the + throughput on rotational devices and on non-queueing flash-based + devices, if processes do synchronous and sequential I/O. In + addition, under BFQ, device idling is also instrumental in + guaranteeing the desired throughput fraction to processes + issuing sync requests (see the description of the slice_idle + tunable in this document, or [1, 2], for more details). + + - With respect to idling for service guarantees, if several + processes are competing for the device at the same time, but + all processes and groups have the same weight, then BFQ + guarantees the expected throughput distribution without ever + idling the device. Throughput is thus as high as possible in + this common scenario. + + - On flash-based storage with internal queueing of commands + (typically NCQ), device idling happens to be always detrimental + for throughput. So, with these devices, BFQ performs idling + only when strictly needed for service guarantees, i.e., for + guaranteeing low latency or fairness. In these cases, overall + throughput may be sub-optimal. No solution currently exists to + provide both strong service guarantees and optimal throughput + on devices with internal queueing. + + - If low-latency mode is enabled (default configuration), BFQ + executes some special heuristics to detect interactive and soft + real-time applications (e.g., video or audio players/streamers), + and to reduce their latency. The most important action taken to + achieve this goal is to give to the queues associated with these + applications more than their fair share of the device + throughput. For brevity, we call just "weight-raising" the whole + sets of actions taken by BFQ to privilege these queues. In + particular, BFQ provides a milder form of weight-raising for + interactive applications, and a stronger form for soft real-time + applications. + + - BFQ automatically deactivates idling for queues born in a burst of + queue creations. In fact, these queues are usually associated with + the processes of applications and services that benefit mostly + from a high throughput. Examples are systemd during boot, or git + grep. + + - As CFQ, BFQ merges queues performing interleaved I/O, i.e., + performing random I/O that becomes mostly sequential if + merged. Differently from CFQ, BFQ achieves this goal with a more + reactive mechanism, called Early Queue Merge (EQM). EQM is so + responsive in detecting interleaved I/O (cooperating processes), + that it enables BFQ to achieve a high throughput, by queue + merging, even for queues for which CFQ needs a different + mechanism, preemption, to get a high throughput. As such EQM is a + unified mechanism to achieve a high throughput with interleaved + I/O. + + - Queues are scheduled according to a variant of WF2Q+, named + B-WF2Q+, and implemented using an augmented rb-tree to preserve an + O(log N) overall complexity. See [2] for more details. B-WF2Q+ is + also ready for hierarchical scheduling, details in Section 4. + + - B-WF2Q+ guarantees a tight deviation with respect to an ideal, + perfectly fair, and smooth service. In particular, B-WF2Q+ + guarantees that each queue receives a fraction of the device + throughput proportional to its weight, even if the throughput + fluctuates, and regardless of: the device parameters, the current + workload and the budgets assigned to the queue. + + - The last, budget-independence, property (although probably + counterintuitive in the first place) is definitely beneficial, for + the following reasons: + + - First, with any proportional-share scheduler, the maximum + deviation with respect to an ideal service is proportional to + the maximum budget (slice) assigned to queues. As a consequence, + BFQ can keep this deviation tight not only because of the + accurate service of B-WF2Q+, but also because BFQ *does not* + need to assign a larger budget to a queue to let the queue + receive a higher fraction of the device throughput. + + - Second, BFQ is free to choose, for every process (queue), the + budget that best fits the needs of the process, or best + leverages the I/O pattern of the process. In particular, BFQ + updates queue budgets with a simple feedback-loop algorithm that + allows a high throughput to be achieved, while still providing + tight latency guarantees to time-sensitive applications. When + the in-service queue expires, this algorithm computes the next + budget of the queue so as to: + + - Let large budgets be eventually assigned to the queues + associated with I/O-bound applications performing sequential + I/O: in fact, the longer these applications are served once + got access to the device, the higher the throughput is. + + - Let small budgets be eventually assigned to the queues + associated with time-sensitive applications (which typically + perform sporadic and short I/O), because, the smaller the + budget assigned to a queue waiting for service is, the sooner + B-WF2Q+ will serve that queue (Subsec 3.3 in [2]). + +- If several processes are competing for the device at the same time, + but all processes and groups have the same weight, then BFQ + guarantees the expected throughput distribution without ever idling + the device. It uses preemption instead. Throughput is then much + higher in this common scenario. + +- ioprio classes are served in strict priority order, i.e., + lower-priority queues are not served as long as there are + higher-priority queues. Among queues in the same class, the + bandwidth is distributed in proportion to the weight of each + queue. A very thin extra bandwidth is however guaranteed to + the Idle class, to prevent it from starving. + + +3. What are BFQ's tunables and how to properly configure BFQ? +============================================================= + +Most BFQ tunables affect service guarantees (basically latency and +fairness) and throughput. For full details on how to choose the +desired tradeoff between service guarantees and throughput, see the +parameters slice_idle, strict_guarantees and low_latency. For details +on how to maximise throughput, see slice_idle, timeout_sync and +max_budget. The other performance-related parameters have been +inherited from, and have been preserved mostly for compatibility with +CFQ. So far, no performance improvement has been reported after +changing the latter parameters in BFQ. + +In particular, the tunables back_seek-max, back_seek_penalty, +fifo_expire_async and fifo_expire_sync below are the same as in +CFQ. Their description is just copied from that for CFQ. Some +considerations in the description of slice_idle are copied from CFQ +too. + +per-process ioprio and weight +----------------------------- + +Unless the cgroups interface is used (see "4. BFQ group scheduling"), +weights can be assigned to processes only indirectly, through I/O +priorities, and according to the relation: +weight = (IOPRIO_BE_NR - ioprio) * 10. + +Beware that, if low-latency is set, then BFQ automatically raises the +weight of the queues associated with interactive and soft real-time +applications. Unset this tunable if you need/want to control weights. + +slice_idle +---------- + +This parameter specifies how long BFQ should idle for next I/O +request, when certain sync BFQ queues become empty. By default +slice_idle is a non-zero value. Idling has a double purpose: boosting +throughput and making sure that the desired throughput distribution is +respected (see the description of how BFQ works, and, if needed, the +papers referred there). + +As for throughput, idling can be very helpful on highly seeky media +like single spindle SATA/SAS disks where we can cut down on overall +number of seeks and see improved throughput. + +Setting slice_idle to 0 will remove all the idling on queues and one +should see an overall improved throughput on faster storage devices +like multiple SATA/SAS disks in hardware RAID configuration, as well +as flash-based storage with internal command queueing (and +parallelism). + +So depending on storage and workload, it might be useful to set +slice_idle=0. In general for SATA/SAS disks and software RAID of +SATA/SAS disks keeping slice_idle enabled should be useful. For any +configurations where there are multiple spindles behind single LUN +(Host based hardware RAID controller or for storage arrays), or with +flash-based fast storage, setting slice_idle=0 might end up in better +throughput and acceptable latencies. + +Idling is however necessary to have service guarantees enforced in +case of differentiated weights or differentiated I/O-request lengths. +To see why, suppose that a given BFQ queue A must get several I/O +requests served for each request served for another queue B. Idling +ensures that, if A makes a new I/O request slightly after becoming +empty, then no request of B is dispatched in the middle, and thus A +does not lose the possibility to get more than one request dispatched +before the next request of B is dispatched. Note that idling +guarantees the desired differentiated treatment of queues only in +terms of I/O-request dispatches. To guarantee that the actual service +order then corresponds to the dispatch order, the strict_guarantees +tunable must be set too. + +There is an important flipside for idling: apart from the above cases +where it is beneficial also for throughput, idling can severely impact +throughput. One important case is random workload. Because of this +issue, BFQ tends to avoid idling as much as possible, when it is not +beneficial also for throughput (as detailed in Section 2). As a +consequence of this behavior, and of further issues described for the +strict_guarantees tunable, short-term service guarantees may be +occasionally violated. And, in some cases, these guarantees may be +more important than guaranteeing maximum throughput. For example, in +video playing/streaming, a very low drop rate may be more important +than maximum throughput. In these cases, consider setting the +strict_guarantees parameter. + +strict_guarantees +----------------- + +If this parameter is set (default: unset), then BFQ + +- always performs idling when the in-service queue becomes empty; + +- forces the device to serve one I/O request at a time, by dispatching a + new request only if there is no outstanding request. + +In the presence of differentiated weights or I/O-request sizes, both +the above conditions are needed to guarantee that every BFQ queue +receives its allotted share of the bandwidth. The first condition is +needed for the reasons explained in the description of the slice_idle +tunable. The second condition is needed because all modern storage +devices reorder internally-queued requests, which may trivially break +the service guarantees enforced by the I/O scheduler. + +Setting strict_guarantees may evidently affect throughput. + +back_seek_max +------------- + +This specifies, given in Kbytes, the maximum "distance" for backward seeking. +The distance is the amount of space from the current head location to the +sectors that are backward in terms of distance. + +This parameter allows the scheduler to anticipate requests in the "backward" +direction and consider them as being the "next" if they are within this +distance from the current head location. + +back_seek_penalty +----------------- + +This parameter is used to compute the cost of backward seeking. If the +backward distance of request is just 1/back_seek_penalty from a "front" +request, then the seeking cost of two requests is considered equivalent. + +So scheduler will not bias toward one or the other request (otherwise scheduler +will bias toward front request). Default value of back_seek_penalty is 2. + +fifo_expire_async +----------------- + +This parameter is used to set the timeout of asynchronous requests. Default +value of this is 248ms. + +fifo_expire_sync +---------------- + +This parameter is used to set the timeout of synchronous requests. Default +value of this is 124ms. In case to favor synchronous requests over asynchronous +one, this value should be decreased relative to fifo_expire_async. + +low_latency +----------- + +This parameter is used to enable/disable BFQ's low latency mode. By +default, low latency mode is enabled. If enabled, interactive and soft +real-time applications are privileged and experience a lower latency, +as explained in more detail in the description of how BFQ works. + +DISABLE this mode if you need full control on bandwidth +distribution. In fact, if it is enabled, then BFQ automatically +increases the bandwidth share of privileged applications, as the main +means to guarantee a lower latency to them. + +In addition, as already highlighted at the beginning of this document, +DISABLE this mode if your only goal is to achieve a high throughput. +In fact, privileging the I/O of some application over the rest may +entail a lower throughput. To achieve the highest-possible throughput +on a non-rotational device, setting slice_idle to 0 may be needed too +(at the cost of giving up any strong guarantee on fairness and low +latency). + +timeout_sync +------------ + +Maximum amount of device time that can be given to a task (queue) once +it has been selected for service. On devices with costly seeks, +increasing this time usually increases maximum throughput. On the +opposite end, increasing this time coarsens the granularity of the +short-term bandwidth and latency guarantees, especially if the +following parameter is set to zero. + +max_budget +---------- + +Maximum amount of service, measured in sectors, that can be provided +to a BFQ queue once it is set in service (of course within the limits +of the above timeout). According to what said in the description of +the algorithm, larger values increase the throughput in proportion to +the percentage of sequential I/O requests issued. The price of larger +values is that they coarsen the granularity of short-term bandwidth +and latency guarantees. + +The default value is 0, which enables auto-tuning: BFQ sets max_budget +to the maximum number of sectors that can be served during +timeout_sync, according to the estimated peak rate. + +For specific devices, some users have occasionally reported to have +reached a higher throughput by setting max_budget explicitly, i.e., by +setting max_budget to a higher value than 0. In particular, they have +set max_budget to higher values than those to which BFQ would have set +it with auto-tuning. An alternative way to achieve this goal is to +just increase the value of timeout_sync, leaving max_budget equal to 0. + +weights +------- + +Read-only parameter, used to show the weights of the currently active +BFQ queues. + + +4. Group scheduling with BFQ +============================ + +BFQ supports both cgroups-v1 and cgroups-v2 io controllers, namely +blkio and io. In particular, BFQ supports weight-based proportional +share. To activate cgroups support, set BFQ_GROUP_IOSCHED. + +4-1 Service guarantees provided +------------------------------- + +With BFQ, proportional share means true proportional share of the +device bandwidth, according to group weights. For example, a group +with weight 200 gets twice the bandwidth, and not just twice the time, +of a group with weight 100. + +BFQ supports hierarchies (group trees) of any depth. Bandwidth is +distributed among groups and processes in the expected way: for each +group, the children of the group share the whole bandwidth of the +group in proportion to their weights. In particular, this implies +that, for each leaf group, every process of the group receives the +same share of the whole group bandwidth, unless the ioprio of the +process is modified. + +The resource-sharing guarantee for a group may partially or totally +switch from bandwidth to time, if providing bandwidth guarantees to +the group lowers the throughput too much. This switch occurs on a +per-process basis: if a process of a leaf group causes throughput loss +if served in such a way to receive its share of the bandwidth, then +BFQ switches back to just time-based proportional share for that +process. + +4-2 Interface +------------- + +To get proportional sharing of bandwidth with BFQ for a given device, +BFQ must of course be the active scheduler for that device. + +Within each group directory, the names of the files associated with +BFQ-specific cgroup parameters and stats begin with the "bfq." +prefix. So, with cgroups-v1 or cgroups-v2, the full prefix for +BFQ-specific files is "blkio.bfq." or "io.bfq." For example, the group +parameter to set the weight of a group with BFQ is blkio.bfq.weight +or io.bfq.weight. + +As for cgroups-v1 (blkio controller), the exact set of stat files +created, and kept up-to-date by bfq, depends on whether +CONFIG_DEBUG_BLK_CGROUP is set. If it is set, then bfq creates all +the stat files documented in +Documentation/cgroup-v1/blkio-controller.txt. If, instead, +CONFIG_DEBUG_BLK_CGROUP is not set, then bfq creates only the files +blkio.bfq.io_service_bytes +blkio.bfq.io_service_bytes_recursive +blkio.bfq.io_serviced +blkio.bfq.io_serviced_recursive + +The value of CONFIG_DEBUG_BLK_CGROUP greatly influences the maximum +throughput sustainable with bfq, because updating the blkio.bfq.* +stats is rather costly, especially for some of the stats enabled by +CONFIG_DEBUG_BLK_CGROUP. + +Parameters to set +----------------- + +For each group, there is only the following parameter to set. + +weight (namely blkio.bfq.weight or io.bfq-weight): the weight of the +group inside its parent. Available values: 1..10000 (default 100). The +linear mapping between ioprio and weights, described at the beginning +of the tunable section, is still valid, but all weights higher than +IOPRIO_BE_NR*10 are mapped to ioprio 0. + +Recall that, if low-latency is set, then BFQ automatically raises the +weight of the queues associated with interactive and soft real-time +applications. Unset this tunable if you need/want to control weights. + + +[1] P. Valente, A. Avanzini, "Evolution of the BFQ Storage I/O + Scheduler", Proceedings of the First Workshop on Mobile System + Technologies (MST-2015), May 2015. + http://algogroup.unimore.it/people/paolo/disk_sched/mst-2015.pdf + +[2] P. Valente and M. Andreolini, "Improving Application + Responsiveness with the BFQ Disk I/O Scheduler", Proceedings of + the 5th Annual International Systems and Storage Conference + (SYSTOR '12), June 2012. + Slightly extended version: + http://algogroup.unimore.it/people/paolo/disk_sched/bfq-v1-suite- + results.pdf diff --git a/Documentation/block/biodoc.txt b/Documentation/block/biodoc.txt new file mode 100644 index 000000000..207eca58e --- /dev/null +++ b/Documentation/block/biodoc.txt @@ -0,0 +1,1165 @@ + Notes on the Generic Block Layer Rewrite in Linux 2.5 + ===================================================== + +Notes Written on Jan 15, 2002: + Jens Axboe <jens.axboe@oracle.com> + Suparna Bhattacharya <suparna@in.ibm.com> + +Last Updated May 2, 2002 +September 2003: Updated I/O Scheduler portions + Nick Piggin <npiggin@kernel.dk> + +Introduction: + +These are some notes describing some aspects of the 2.5 block layer in the +context of the bio rewrite. The idea is to bring out some of the key +changes and a glimpse of the rationale behind those changes. + +Please mail corrections & suggestions to suparna@in.ibm.com. + +Credits: +--------- + +2.5 bio rewrite: + Jens Axboe <jens.axboe@oracle.com> + +Many aspects of the generic block layer redesign were driven by and evolved +over discussions, prior patches and the collective experience of several +people. See sections 8 and 9 for a list of some related references. + +The following people helped with review comments and inputs for this +document: + Christoph Hellwig <hch@infradead.org> + Arjan van de Ven <arjanv@redhat.com> + Randy Dunlap <rdunlap@xenotime.net> + Andre Hedrick <andre@linux-ide.org> + +The following people helped with fixes/contributions to the bio patches +while it was still work-in-progress: + David S. Miller <davem@redhat.com> + + +Description of Contents: +------------------------ + +1. Scope for tuning of logic to various needs + 1.1 Tuning based on device or low level driver capabilities + - Per-queue parameters + - Highmem I/O support + - I/O scheduler modularization + 1.2 Tuning based on high level requirements/capabilities + 1.2.1 Request Priority/Latency + 1.3 Direct access/bypass to lower layers for diagnostics and special + device operations + 1.3.1 Pre-built commands +2. New flexible and generic but minimalist i/o structure or descriptor + (instead of using buffer heads at the i/o layer) + 2.1 Requirements/Goals addressed + 2.2 The bio struct in detail (multi-page io unit) + 2.3 Changes in the request structure +3. Using bios + 3.1 Setup/teardown (allocation, splitting) + 3.2 Generic bio helper routines + 3.2.1 Traversing segments and completion units in a request + 3.2.2 Setting up DMA scatterlists + 3.2.3 I/O completion + 3.2.4 Implications for drivers that do not interpret bios (don't handle + multiple segments) + 3.2.5 Request command tagging + 3.3 I/O submission +4. The I/O scheduler +5. Scalability related changes + 5.1 Granular locking: Removal of io_request_lock + 5.2 Prepare for transition to 64 bit sector_t +6. Other Changes/Implications + 6.1 Partition re-mapping handled by the generic block layer +7. A few tips on migration of older drivers +8. A list of prior/related/impacted patches/ideas +9. Other References/Discussion Threads + +--------------------------------------------------------------------------- + +Bio Notes +-------- + +Let us discuss the changes in the context of how some overall goals for the +block layer are addressed. + +1. Scope for tuning the generic logic to satisfy various requirements + +The block layer design supports adaptable abstractions to handle common +processing with the ability to tune the logic to an appropriate extent +depending on the nature of the device and the requirements of the caller. +One of the objectives of the rewrite was to increase the degree of tunability +and to enable higher level code to utilize underlying device/driver +capabilities to the maximum extent for better i/o performance. This is +important especially in the light of ever improving hardware capabilities +and application/middleware software designed to take advantage of these +capabilities. + +1.1 Tuning based on low level device / driver capabilities + +Sophisticated devices with large built-in caches, intelligent i/o scheduling +optimizations, high memory DMA support, etc may find some of the +generic processing an overhead, while for less capable devices the +generic functionality is essential for performance or correctness reasons. +Knowledge of some of the capabilities or parameters of the device should be +used at the generic block layer to take the right decisions on +behalf of the driver. + +How is this achieved ? + +Tuning at a per-queue level: + +i. Per-queue limits/values exported to the generic layer by the driver + +Various parameters that the generic i/o scheduler logic uses are set at +a per-queue level (e.g maximum request size, maximum number of segments in +a scatter-gather list, logical block size) + +Some parameters that were earlier available as global arrays indexed by +major/minor are now directly associated with the queue. Some of these may +move into the block device structure in the future. Some characteristics +have been incorporated into a queue flags field rather than separate fields +in themselves. There are blk_queue_xxx functions to set the parameters, +rather than update the fields directly + +Some new queue property settings: + + blk_queue_bounce_limit(q, u64 dma_address) + Enable I/O to highmem pages, dma_address being the + limit. No highmem default. + + blk_queue_max_sectors(q, max_sectors) + Sets two variables that limit the size of the request. + + - The request queue's max_sectors, which is a soft size in + units of 512 byte sectors, and could be dynamically varied + by the core kernel. + + - The request queue's max_hw_sectors, which is a hard limit + and reflects the maximum size request a driver can handle + in units of 512 byte sectors. + + The default for both max_sectors and max_hw_sectors is + 255. The upper limit of max_sectors is 1024. + + blk_queue_max_phys_segments(q, max_segments) + Maximum physical segments you can handle in a request. 128 + default (driver limit). (See 3.2.2) + + blk_queue_max_hw_segments(q, max_segments) + Maximum dma segments the hardware can handle in a request. 128 + default (host adapter limit, after dma remapping). + (See 3.2.2) + + blk_queue_max_segment_size(q, max_seg_size) + Maximum size of a clustered segment, 64kB default. + + blk_queue_logical_block_size(q, logical_block_size) + Lowest possible sector size that the hardware can operate + on, 512 bytes default. + +New queue flags: + + QUEUE_FLAG_CLUSTER (see 3.2.2) + QUEUE_FLAG_QUEUED (see 3.2.4) + + +ii. High-mem i/o capabilities are now considered the default + +The generic bounce buffer logic, present in 2.4, where the block layer would +by default copyin/out i/o requests on high-memory buffers to low-memory buffers +assuming that the driver wouldn't be able to handle it directly, has been +changed in 2.5. The bounce logic is now applied only for memory ranges +for which the device cannot handle i/o. A driver can specify this by +setting the queue bounce limit for the request queue for the device +(blk_queue_bounce_limit()). This avoids the inefficiencies of the copyin/out +where a device is capable of handling high memory i/o. + +In order to enable high-memory i/o where the device is capable of supporting +it, the pci dma mapping routines and associated data structures have now been +modified to accomplish a direct page -> bus translation, without requiring +a virtual address mapping (unlike the earlier scheme of virtual address +-> bus translation). So this works uniformly for high-memory pages (which +do not have a corresponding kernel virtual address space mapping) and +low-memory pages. + +Note: Please refer to Documentation/DMA-API-HOWTO.txt for a discussion +on PCI high mem DMA aspects and mapping of scatter gather lists, and support +for 64 bit PCI. + +Special handling is required only for cases where i/o needs to happen on +pages at physical memory addresses beyond what the device can support. In these +cases, a bounce bio representing a buffer from the supported memory range +is used for performing the i/o with copyin/copyout as needed depending on +the type of the operation. For example, in case of a read operation, the +data read has to be copied to the original buffer on i/o completion, so a +callback routine is set up to do this, while for write, the data is copied +from the original buffer to the bounce buffer prior to issuing the +operation. Since an original buffer may be in a high memory area that's not +mapped in kernel virtual addr, a kmap operation may be required for +performing the copy, and special care may be needed in the completion path +as it may not be in irq context. Special care is also required (by way of +GFP flags) when allocating bounce buffers, to avoid certain highmem +deadlock possibilities. + +It is also possible that a bounce buffer may be allocated from high-memory +area that's not mapped in kernel virtual addr, but within the range that the +device can use directly; so the bounce page may need to be kmapped during +copy operations. [Note: This does not hold in the current implementation, +though] + +There are some situations when pages from high memory may need to +be kmapped, even if bounce buffers are not necessary. For example a device +may need to abort DMA operations and revert to PIO for the transfer, in +which case a virtual mapping of the page is required. For SCSI it is also +done in some scenarios where the low level driver cannot be trusted to +handle a single sg entry correctly. The driver is expected to perform the +kmaps as needed on such occasions as appropriate. A driver could also use +the blk_queue_bounce() routine on its own to bounce highmem i/o to low +memory for specific requests if so desired. + +iii. The i/o scheduler algorithm itself can be replaced/set as appropriate + +As in 2.4, it is possible to plugin a brand new i/o scheduler for a particular +queue or pick from (copy) existing generic schedulers and replace/override +certain portions of it. The 2.5 rewrite provides improved modularization +of the i/o scheduler. There are more pluggable callbacks, e.g for init, +add request, extract request, which makes it possible to abstract specific +i/o scheduling algorithm aspects and details outside of the generic loop. +It also makes it possible to completely hide the implementation details of +the i/o scheduler from block drivers. + +I/O scheduler wrappers are to be used instead of accessing the queue directly. +See section 4. The I/O scheduler for details. + +1.2 Tuning Based on High level code capabilities + +i. Application capabilities for raw i/o + +This comes from some of the high-performance database/middleware +requirements where an application prefers to make its own i/o scheduling +decisions based on an understanding of the access patterns and i/o +characteristics + +ii. High performance filesystems or other higher level kernel code's +capabilities + +Kernel components like filesystems could also take their own i/o scheduling +decisions for optimizing performance. Journalling filesystems may need +some control over i/o ordering. + +What kind of support exists at the generic block layer for this ? + +The flags and rw fields in the bio structure can be used for some tuning +from above e.g indicating that an i/o is just a readahead request, or priority +settings (currently unused). As far as user applications are concerned they +would need an additional mechanism either via open flags or ioctls, or some +other upper level mechanism to communicate such settings to block. + +1.2.1 Request Priority/Latency + +Todo/Under discussion: +Arjan's proposed request priority scheme allows higher levels some broad + control (high/med/low) over the priority of an i/o request vs other pending + requests in the queue. For example it allows reads for bringing in an + executable page on demand to be given a higher priority over pending write + requests which haven't aged too much on the queue. Potentially this priority + could even be exposed to applications in some manner, providing higher level + tunability. Time based aging avoids starvation of lower priority + requests. Some bits in the bi_opf flags field in the bio structure are + intended to be used for this priority information. + + +1.3 Direct Access to Low level Device/Driver Capabilities (Bypass mode) + (e.g Diagnostics, Systems Management) + +There are situations where high-level code needs to have direct access to +the low level device capabilities or requires the ability to issue commands +to the device bypassing some of the intermediate i/o layers. +These could, for example, be special control commands issued through ioctl +interfaces, or could be raw read/write commands that stress the drive's +capabilities for certain kinds of fitness tests. Having direct interfaces at +multiple levels without having to pass through upper layers makes +it possible to perform bottom up validation of the i/o path, layer by +layer, starting from the media. + +The normal i/o submission interfaces, e.g submit_bio, could be bypassed +for specially crafted requests which such ioctl or diagnostics +interfaces would typically use, and the elevator add_request routine +can instead be used to directly insert such requests in the queue or preferably +the blk_do_rq routine can be used to place the request on the queue and +wait for completion. Alternatively, sometimes the caller might just +invoke a lower level driver specific interface with the request as a +parameter. + +If the request is a means for passing on special information associated with +the command, then such information is associated with the request->special +field (rather than misuse the request->buffer field which is meant for the +request data buffer's virtual mapping). + +For passing request data, the caller must build up a bio descriptor +representing the concerned memory buffer if the underlying driver interprets +bio segments or uses the block layer end*request* functions for i/o +completion. Alternatively one could directly use the request->buffer field to +specify the virtual address of the buffer, if the driver expects buffer +addresses passed in this way and ignores bio entries for the request type +involved. In the latter case, the driver would modify and manage the +request->buffer, request->sector and request->nr_sectors or +request->current_nr_sectors fields itself rather than using the block layer +end_request or end_that_request_first completion interfaces. +(See 2.3 or Documentation/block/request.txt for a brief explanation of +the request structure fields) + +[TBD: end_that_request_last should be usable even in this case; +Perhaps an end_that_direct_request_first routine could be implemented to make +handling direct requests easier for such drivers; Also for drivers that +expect bios, a helper function could be provided for setting up a bio +corresponding to a data buffer] + +<JENS: I dont understand the above, why is end_that_request_first() not +usable? Or _last for that matter. I must be missing something> +<SUP: What I meant here was that if the request doesn't have a bio, then + end_that_request_first doesn't modify nr_sectors or current_nr_sectors, + and hence can't be used for advancing request state settings on the + completion of partial transfers. The driver has to modify these fields + directly by hand. + This is because end_that_request_first only iterates over the bio list, + and always returns 0 if there are none associated with the request. + _last works OK in this case, and is not a problem, as I mentioned earlier +> + +1.3.1 Pre-built Commands + +A request can be created with a pre-built custom command to be sent directly +to the device. The cmd block in the request structure has room for filling +in the command bytes. (i.e rq->cmd is now 16 bytes in size, and meant for +command pre-building, and the type of the request is now indicated +through rq->flags instead of via rq->cmd) + +The request structure flags can be set up to indicate the type of request +in such cases (REQ_PC: direct packet command passed to driver, REQ_BLOCK_PC: +packet command issued via blk_do_rq, REQ_SPECIAL: special request). + +It can help to pre-build device commands for requests in advance. +Drivers can now specify a request prepare function (q->prep_rq_fn) that the +block layer would invoke to pre-build device commands for a given request, +or perform other preparatory processing for the request. This is routine is +called by elv_next_request(), i.e. typically just before servicing a request. +(The prepare function would not be called for requests that have RQF_DONTPREP +enabled) + +Aside: + Pre-building could possibly even be done early, i.e before placing the + request on the queue, rather than construct the command on the fly in the + driver while servicing the request queue when it may affect latencies in + interrupt context or responsiveness in general. One way to add early + pre-building would be to do it whenever we fail to merge on a request. + Now REQ_NOMERGE is set in the request flags to skip this one in the future, + which means that it will not change before we feed it to the device. So + the pre-builder hook can be invoked there. + + +2. Flexible and generic but minimalist i/o structure/descriptor. + +2.1 Reason for a new structure and requirements addressed + +Prior to 2.5, buffer heads were used as the unit of i/o at the generic block +layer, and the low level request structure was associated with a chain of +buffer heads for a contiguous i/o request. This led to certain inefficiencies +when it came to large i/o requests and readv/writev style operations, as it +forced such requests to be broken up into small chunks before being passed +on to the generic block layer, only to be merged by the i/o scheduler +when the underlying device was capable of handling the i/o in one shot. +Also, using the buffer head as an i/o structure for i/os that didn't originate +from the buffer cache unnecessarily added to the weight of the descriptors +which were generated for each such chunk. + +The following were some of the goals and expectations considered in the +redesign of the block i/o data structure in 2.5. + +i. Should be appropriate as a descriptor for both raw and buffered i/o - + avoid cache related fields which are irrelevant in the direct/page i/o path, + or filesystem block size alignment restrictions which may not be relevant + for raw i/o. +ii. Ability to represent high-memory buffers (which do not have a virtual + address mapping in kernel address space). +iii.Ability to represent large i/os w/o unnecessarily breaking them up (i.e + greater than PAGE_SIZE chunks in one shot) +iv. At the same time, ability to retain independent identity of i/os from + different sources or i/o units requiring individual completion (e.g. for + latency reasons) +v. Ability to represent an i/o involving multiple physical memory segments + (including non-page aligned page fragments, as specified via readv/writev) + without unnecessarily breaking it up, if the underlying device is capable of + handling it. +vi. Preferably should be based on a memory descriptor structure that can be + passed around different types of subsystems or layers, maybe even + networking, without duplication or extra copies of data/descriptor fields + themselves in the process +vii.Ability to handle the possibility of splits/merges as the structure passes + through layered drivers (lvm, md, evms), with minimal overhead. + +The solution was to define a new structure (bio) for the block layer, +instead of using the buffer head structure (bh) directly, the idea being +avoidance of some associated baggage and limitations. The bio structure +is uniformly used for all i/o at the block layer ; it forms a part of the +bh structure for buffered i/o, and in the case of raw/direct i/o kiobufs are +mapped to bio structures. + +2.2 The bio struct + +The bio structure uses a vector representation pointing to an array of tuples +of <page, offset, len> to describe the i/o buffer, and has various other +fields describing i/o parameters and state that needs to be maintained for +performing the i/o. + +Notice that this representation means that a bio has no virtual address +mapping at all (unlike buffer heads). + +struct bio_vec { + struct page *bv_page; + unsigned short bv_len; + unsigned short bv_offset; +}; + +/* + * main unit of I/O for the block layer and lower layers (ie drivers) + */ +struct bio { + struct bio *bi_next; /* request queue link */ + struct block_device *bi_bdev; /* target device */ + unsigned long bi_flags; /* status, command, etc */ + unsigned long bi_opf; /* low bits: r/w, high: priority */ + + unsigned int bi_vcnt; /* how may bio_vec's */ + struct bvec_iter bi_iter; /* current index into bio_vec array */ + + unsigned int bi_size; /* total size in bytes */ + unsigned short bi_phys_segments; /* segments after physaddr coalesce*/ + unsigned short bi_hw_segments; /* segments after DMA remapping */ + unsigned int bi_max; /* max bio_vecs we can hold + used as index into pool */ + struct bio_vec *bi_io_vec; /* the actual vec list */ + bio_end_io_t *bi_end_io; /* bi_end_io (bio) */ + atomic_t bi_cnt; /* pin count: free when it hits zero */ + void *bi_private; +}; + +With this multipage bio design: + +- Large i/os can be sent down in one go using a bio_vec list consisting + of an array of <page, offset, len> fragments (similar to the way fragments + are represented in the zero-copy network code) +- Splitting of an i/o request across multiple devices (as in the case of + lvm or raid) is achieved by cloning the bio (where the clone points to + the same bi_io_vec array, but with the index and size accordingly modified) +- A linked list of bios is used as before for unrelated merges (*) - this + avoids reallocs and makes independent completions easier to handle. +- Code that traverses the req list can find all the segments of a bio + by using rq_for_each_segment. This handles the fact that a request + has multiple bios, each of which can have multiple segments. +- Drivers which can't process a large bio in one shot can use the bi_iter + field to keep track of the next bio_vec entry to process. + (e.g a 1MB bio_vec needs to be handled in max 128kB chunks for IDE) + [TBD: Should preferably also have a bi_voffset and bi_vlen to avoid modifying + bi_offset an len fields] + +(*) unrelated merges -- a request ends up containing two or more bios that + didn't originate from the same place. + +bi_end_io() i/o callback gets called on i/o completion of the entire bio. + +At a lower level, drivers build a scatter gather list from the merged bios. +The scatter gather list is in the form of an array of <page, offset, len> +entries with their corresponding dma address mappings filled in at the +appropriate time. As an optimization, contiguous physical pages can be +covered by a single entry where <page> refers to the first page and <len> +covers the range of pages (up to 16 contiguous pages could be covered this +way). There is a helper routine (blk_rq_map_sg) which drivers can use to build +the sg list. + +Note: Right now the only user of bios with more than one page is ll_rw_kio, +which in turn means that only raw I/O uses it (direct i/o may not work +right now). The intent however is to enable clustering of pages etc to +become possible. The pagebuf abstraction layer from SGI also uses multi-page +bios, but that is currently not included in the stock development kernels. +The same is true of Andrew Morton's work-in-progress multipage bio writeout +and readahead patches. + +2.3 Changes in the Request Structure + +The request structure is the structure that gets passed down to low level +drivers. The block layer make_request function builds up a request structure, +places it on the queue and invokes the drivers request_fn. The driver makes +use of block layer helper routine elv_next_request to pull the next request +off the queue. Control or diagnostic functions might bypass block and directly +invoke underlying driver entry points passing in a specially constructed +request structure. + +Only some relevant fields (mainly those which changed or may be referred +to in some of the discussion here) are listed below, not necessarily in +the order in which they occur in the structure (see include/linux/blkdev.h) +Refer to Documentation/block/request.txt for details about all the request +structure fields and a quick reference about the layers which are +supposed to use or modify those fields. + +struct request { + struct list_head queuelist; /* Not meant to be directly accessed by + the driver. + Used by q->elv_next_request_fn + rq->queue is gone + */ + . + . + unsigned char cmd[16]; /* prebuilt command data block */ + unsigned long flags; /* also includes earlier rq->cmd settings */ + . + . + sector_t sector; /* this field is now of type sector_t instead of int + preparation for 64 bit sectors */ + . + . + + /* Number of scatter-gather DMA addr+len pairs after + * physical address coalescing is performed. + */ + unsigned short nr_phys_segments; + + /* Number of scatter-gather addr+len pairs after + * physical and DMA remapping hardware coalescing is performed. + * This is the number of scatter-gather entries the driver + * will actually have to deal with after DMA mapping is done. + */ + unsigned short nr_hw_segments; + + /* Various sector counts */ + unsigned long nr_sectors; /* no. of sectors left: driver modifiable */ + unsigned long hard_nr_sectors; /* block internal copy of above */ + unsigned int current_nr_sectors; /* no. of sectors left in the + current segment:driver modifiable */ + unsigned long hard_cur_sectors; /* block internal copy of the above */ + . + . + int tag; /* command tag associated with request */ + void *special; /* same as before */ + char *buffer; /* valid only for low memory buffers up to + current_nr_sectors */ + . + . + struct bio *bio, *biotail; /* bio list instead of bh */ + struct request_list *rl; +} + +See the req_ops and req_flag_bits definitions for an explanation of the various +flags available. Some bits are used by the block layer or i/o scheduler. + +The behaviour of the various sector counts are almost the same as before, +except that since we have multi-segment bios, current_nr_sectors refers +to the numbers of sectors in the current segment being processed which could +be one of the many segments in the current bio (i.e i/o completion unit). +The nr_sectors value refers to the total number of sectors in the whole +request that remain to be transferred (no change). The purpose of the +hard_xxx values is for block to remember these counts every time it hands +over the request to the driver. These values are updated by block on +end_that_request_first, i.e. every time the driver completes a part of the +transfer and invokes block end*request helpers to mark this. The +driver should not modify these values. The block layer sets up the +nr_sectors and current_nr_sectors fields (based on the corresponding +hard_xxx values and the number of bytes transferred) and updates it on +every transfer that invokes end_that_request_first. It does the same for the +buffer, bio, bio->bi_iter fields too. + +The buffer field is just a virtual address mapping of the current segment +of the i/o buffer in cases where the buffer resides in low-memory. For high +memory i/o, this field is not valid and must not be used by drivers. + +Code that sets up its own request structures and passes them down to +a driver needs to be careful about interoperation with the block layer helper +functions which the driver uses. (Section 1.3) + +3. Using bios + +3.1 Setup/Teardown + +There are routines for managing the allocation, and reference counting, and +freeing of bios (bio_alloc, bio_get, bio_put). + +This makes use of Ingo Molnar's mempool implementation, which enables +subsystems like bio to maintain their own reserve memory pools for guaranteed +deadlock-free allocations during extreme VM load. For example, the VM +subsystem makes use of the block layer to writeout dirty pages in order to be +able to free up memory space, a case which needs careful handling. The +allocation logic draws from the preallocated emergency reserve in situations +where it cannot allocate through normal means. If the pool is empty and it +can wait, then it would trigger action that would help free up memory or +replenish the pool (without deadlocking) and wait for availability in the pool. +If it is in IRQ context, and hence not in a position to do this, allocation +could fail if the pool is empty. In general mempool always first tries to +perform allocation without having to wait, even if it means digging into the +pool as long it is not less that 50% full. + +On a free, memory is released to the pool or directly freed depending on +the current availability in the pool. The mempool interface lets the +subsystem specify the routines to be used for normal alloc and free. In the +case of bio, these routines make use of the standard slab allocator. + +The caller of bio_alloc is expected to taken certain steps to avoid +deadlocks, e.g. avoid trying to allocate more memory from the pool while +already holding memory obtained from the pool. +[TBD: This is a potential issue, though a rare possibility + in the bounce bio allocation that happens in the current code, since + it ends up allocating a second bio from the same pool while + holding the original bio ] + +Memory allocated from the pool should be released back within a limited +amount of time (in the case of bio, that would be after the i/o is completed). +This ensures that if part of the pool has been used up, some work (in this +case i/o) must already be in progress and memory would be available when it +is over. If allocating from multiple pools in the same code path, the order +or hierarchy of allocation needs to be consistent, just the way one deals +with multiple locks. + +The bio_alloc routine also needs to allocate the bio_vec_list (bvec_alloc()) +for a non-clone bio. There are the 6 pools setup for different size biovecs, +so bio_alloc(gfp_mask, nr_iovecs) will allocate a vec_list of the +given size from these slabs. + +The bio_get() routine may be used to hold an extra reference on a bio prior +to i/o submission, if the bio fields are likely to be accessed after the +i/o is issued (since the bio may otherwise get freed in case i/o completion +happens in the meantime). + +The bio_clone_fast() routine may be used to duplicate a bio, where the clone +shares the bio_vec_list with the original bio (i.e. both point to the +same bio_vec_list). This would typically be used for splitting i/o requests +in lvm or md. + +3.2 Generic bio helper Routines + +3.2.1 Traversing segments and completion units in a request + +The macro rq_for_each_segment() should be used for traversing the bios +in the request list (drivers should avoid directly trying to do it +themselves). Using these helpers should also make it easier to cope +with block changes in the future. + + struct req_iterator iter; + rq_for_each_segment(bio_vec, rq, iter) + /* bio_vec is now current segment */ + +I/O completion callbacks are per-bio rather than per-segment, so drivers +that traverse bio chains on completion need to keep that in mind. Drivers +which don't make a distinction between segments and completion units would +need to be reorganized to support multi-segment bios. + +3.2.2 Setting up DMA scatterlists + +The blk_rq_map_sg() helper routine would be used for setting up scatter +gather lists from a request, so a driver need not do it on its own. + + nr_segments = blk_rq_map_sg(q, rq, scatterlist); + +The helper routine provides a level of abstraction which makes it easier +to modify the internals of request to scatterlist conversion down the line +without breaking drivers. The blk_rq_map_sg routine takes care of several +things like collapsing physically contiguous segments (if QUEUE_FLAG_CLUSTER +is set) and correct segment accounting to avoid exceeding the limits which +the i/o hardware can handle, based on various queue properties. + +- Prevents a clustered segment from crossing a 4GB mem boundary +- Avoids building segments that would exceed the number of physical + memory segments that the driver can handle (phys_segments) and the + number that the underlying hardware can handle at once, accounting for + DMA remapping (hw_segments) (i.e. IOMMU aware limits). + +Routines which the low level driver can use to set up the segment limits: + +blk_queue_max_hw_segments() : Sets an upper limit of the maximum number of +hw data segments in a request (i.e. the maximum number of address/length +pairs the host adapter can actually hand to the device at once) + +blk_queue_max_phys_segments() : Sets an upper limit on the maximum number +of physical data segments in a request (i.e. the largest sized scatter list +a driver could handle) + +3.2.3 I/O completion + +The existing generic block layer helper routines end_request, +end_that_request_first and end_that_request_last can be used for i/o +completion (and setting things up so the rest of the i/o or the next +request can be kicked of) as before. With the introduction of multi-page +bio support, end_that_request_first requires an additional argument indicating +the number of sectors completed. + +3.2.4 Implications for drivers that do not interpret bios (don't handle + multiple segments) + +Drivers that do not interpret bios e.g those which do not handle multiple +segments and do not support i/o into high memory addresses (require bounce +buffers) and expect only virtually mapped buffers, can access the rq->buffer +field. As before the driver should use current_nr_sectors to determine the +size of remaining data in the current segment (that is the maximum it can +transfer in one go unless it interprets segments), and rely on the block layer +end_request, or end_that_request_first/last to take care of all accounting +and transparent mapping of the next bio segment when a segment boundary +is crossed on completion of a transfer. (The end*request* functions should +be used if only if the request has come down from block/bio path, not for +direct access requests which only specify rq->buffer without a valid rq->bio) + +3.2.5 Generic request command tagging + +3.2.5.1 Tag helpers + +Block now offers some simple generic functionality to help support command +queueing (typically known as tagged command queueing), ie manage more than +one outstanding command on a queue at any given time. + + blk_queue_init_tags(struct request_queue *q, int depth) + + Initialize internal command tagging structures for a maximum + depth of 'depth'. + + blk_queue_free_tags((struct request_queue *q) + + Teardown tag info associated with the queue. This will be done + automatically by block if blk_queue_cleanup() is called on a queue + that is using tagging. + +The above are initialization and exit management, the main helpers during +normal operations are: + + blk_queue_start_tag(struct request_queue *q, struct request *rq) + + Start tagged operation for this request. A free tag number between + 0 and 'depth' is assigned to the request (rq->tag holds this number), + and 'rq' is added to the internal tag management. If the maximum depth + for this queue is already achieved (or if the tag wasn't started for + some other reason), 1 is returned. Otherwise 0 is returned. + + blk_queue_end_tag(struct request_queue *q, struct request *rq) + + End tagged operation on this request. 'rq' is removed from the internal + book keeping structures. + +To minimize struct request and queue overhead, the tag helpers utilize some +of the same request members that are used for normal request queue management. +This means that a request cannot both be an active tag and be on the queue +list at the same time. blk_queue_start_tag() will remove the request, but +the driver must remember to call blk_queue_end_tag() before signalling +completion of the request to the block layer. This means ending tag +operations before calling end_that_request_last()! For an example of a user +of these helpers, see the IDE tagged command queueing support. + +3.2.5.2 Tag info + +Some block functions exist to query current tag status or to go from a +tag number to the associated request. These are, in no particular order: + + blk_queue_tagged(q) + + Returns 1 if the queue 'q' is using tagging, 0 if not. + + blk_queue_tag_request(q, tag) + + Returns a pointer to the request associated with tag 'tag'. + + blk_queue_tag_depth(q) + + Return current queue depth. + + blk_queue_tag_queue(q) + + Returns 1 if the queue can accept a new queued command, 0 if we are + at the maximum depth already. + + blk_queue_rq_tagged(rq) + + Returns 1 if the request 'rq' is tagged. + +3.2.5.2 Internal structure + +Internally, block manages tags in the blk_queue_tag structure: + + struct blk_queue_tag { + struct request **tag_index; /* array or pointers to rq */ + unsigned long *tag_map; /* bitmap of free tags */ + struct list_head busy_list; /* fifo list of busy tags */ + int busy; /* queue depth */ + int max_depth; /* max queue depth */ + }; + +Most of the above is simple and straight forward, however busy_list may need +a bit of explaining. Normally we don't care too much about request ordering, +but in the event of any barrier requests in the tag queue we need to ensure +that requests are restarted in the order they were queue. + +3.3 I/O Submission + +The routine submit_bio() is used to submit a single io. Higher level i/o +routines make use of this: + +(a) Buffered i/o: +The routine submit_bh() invokes submit_bio() on a bio corresponding to the +bh, allocating the bio if required. ll_rw_block() uses submit_bh() as before. + +(b) Kiobuf i/o (for raw/direct i/o): +The ll_rw_kio() routine breaks up the kiobuf into page sized chunks and +maps the array to one or more multi-page bios, issuing submit_bio() to +perform the i/o on each of these. + +The embedded bh array in the kiobuf structure has been removed and no +preallocation of bios is done for kiobufs. [The intent is to remove the +blocks array as well, but it's currently in there to kludge around direct i/o.] +Thus kiobuf allocation has switched back to using kmalloc rather than vmalloc. + +Todo/Observation: + + A single kiobuf structure is assumed to correspond to a contiguous range + of data, so brw_kiovec() invokes ll_rw_kio for each kiobuf in a kiovec. + So right now it wouldn't work for direct i/o on non-contiguous blocks. + This is to be resolved. The eventual direction is to replace kiobuf + by kvec's. + + Badari Pulavarty has a patch to implement direct i/o correctly using + bio and kvec. + + +(c) Page i/o: +Todo/Under discussion: + + Andrew Morton's multi-page bio patches attempt to issue multi-page + writeouts (and reads) from the page cache, by directly building up + large bios for submission completely bypassing the usage of buffer + heads. This work is still in progress. + + Christoph Hellwig had some code that uses bios for page-io (rather than + bh). This isn't included in bio as yet. Christoph was also working on a + design for representing virtual/real extents as an entity and modifying + some of the address space ops interfaces to utilize this abstraction rather + than buffer_heads. (This is somewhat along the lines of the SGI XFS pagebuf + abstraction, but intended to be as lightweight as possible). + +(d) Direct access i/o: +Direct access requests that do not contain bios would be submitted differently +as discussed earlier in section 1.3. + +Aside: + + Kvec i/o: + + Ben LaHaise's aio code uses a slightly different structure instead + of kiobufs, called a kvec_cb. This contains an array of <page, offset, len> + tuples (very much like the networking code), together with a callback function + and data pointer. This is embedded into a brw_cb structure when passed + to brw_kvec_async(). + + Now it should be possible to directly map these kvecs to a bio. Just as while + cloning, in this case rather than PRE_BUILT bio_vecs, we set the bi_io_vec + array pointer to point to the veclet array in kvecs. + + TBD: In order for this to work, some changes are needed in the way multi-page + bios are handled today. The values of the tuples in such a vector passed in + from higher level code should not be modified by the block layer in the course + of its request processing, since that would make it hard for the higher layer + to continue to use the vector descriptor (kvec) after i/o completes. Instead, + all such transient state should either be maintained in the request structure, + and passed on in some way to the endio completion routine. + + +4. The I/O scheduler +I/O scheduler, a.k.a. elevator, is implemented in two layers. Generic dispatch +queue and specific I/O schedulers. Unless stated otherwise, elevator is used +to refer to both parts and I/O scheduler to specific I/O schedulers. + +Block layer implements generic dispatch queue in block/*.c. +The generic dispatch queue is responsible for requeueing, handling non-fs +requests and all other subtleties. + +Specific I/O schedulers are responsible for ordering normal filesystem +requests. They can also choose to delay certain requests to improve +throughput or whatever purpose. As the plural form indicates, there are +multiple I/O schedulers. They can be built as modules but at least one should +be built inside the kernel. Each queue can choose different one and can also +change to another one dynamically. + +A block layer call to the i/o scheduler follows the convention elv_xxx(). This +calls elevator_xxx_fn in the elevator switch (block/elevator.c). Oh, xxx +and xxx might not match exactly, but use your imagination. If an elevator +doesn't implement a function, the switch does nothing or some minimal house +keeping work. + +4.1. I/O scheduler API + +The functions an elevator may implement are: (* are mandatory) +elevator_merge_fn called to query requests for merge with a bio + +elevator_merge_req_fn called when two requests get merged. the one + which gets merged into the other one will be + never seen by I/O scheduler again. IOW, after + being merged, the request is gone. + +elevator_merged_fn called when a request in the scheduler has been + involved in a merge. It is used in the deadline + scheduler for example, to reposition the request + if its sorting order has changed. + +elevator_allow_merge_fn called whenever the block layer determines + that a bio can be merged into an existing + request safely. The io scheduler may still + want to stop a merge at this point if it + results in some sort of conflict internally, + this hook allows it to do that. Note however + that two *requests* can still be merged at later + time. Currently the io scheduler has no way to + prevent that. It can only learn about the fact + from elevator_merge_req_fn callback. + +elevator_dispatch_fn* fills the dispatch queue with ready requests. + I/O schedulers are free to postpone requests by + not filling the dispatch queue unless @force + is non-zero. Once dispatched, I/O schedulers + are not allowed to manipulate the requests - + they belong to generic dispatch queue. + +elevator_add_req_fn* called to add a new request into the scheduler + +elevator_former_req_fn +elevator_latter_req_fn These return the request before or after the + one specified in disk sort order. Used by the + block layer to find merge possibilities. + +elevator_completed_req_fn called when a request is completed. + +elevator_may_queue_fn returns true if the scheduler wants to allow the + current context to queue a new request even if + it is over the queue limit. This must be used + very carefully!! + +elevator_set_req_fn +elevator_put_req_fn Must be used to allocate and free any elevator + specific storage for a request. + +elevator_activate_req_fn Called when device driver first sees a request. + I/O schedulers can use this callback to + determine when actual execution of a request + starts. +elevator_deactivate_req_fn Called when device driver decides to delay + a request by requeueing it. + +elevator_init_fn* +elevator_exit_fn Allocate and free any elevator specific storage + for a queue. + +4.2 Request flows seen by I/O schedulers +All requests seen by I/O schedulers strictly follow one of the following three +flows. + + set_req_fn -> + + i. add_req_fn -> (merged_fn ->)* -> dispatch_fn -> activate_req_fn -> + (deactivate_req_fn -> activate_req_fn ->)* -> completed_req_fn + ii. add_req_fn -> (merged_fn ->)* -> merge_req_fn + iii. [none] + + -> put_req_fn + +4.3 I/O scheduler implementation +The generic i/o scheduler algorithm attempts to sort/merge/batch requests for +optimal disk scan and request servicing performance (based on generic +principles and device capabilities), optimized for: +i. improved throughput +ii. improved latency +iii. better utilization of h/w & CPU time + +Characteristics: + +i. Binary tree +AS and deadline i/o schedulers use red black binary trees for disk position +sorting and searching, and a fifo linked list for time-based searching. This +gives good scalability and good availability of information. Requests are +almost always dispatched in disk sort order, so a cache is kept of the next +request in sort order to prevent binary tree lookups. + +This arrangement is not a generic block layer characteristic however, so +elevators may implement queues as they please. + +ii. Merge hash +AS and deadline use a hash table indexed by the last sector of a request. This +enables merging code to quickly look up "back merge" candidates, even when +multiple I/O streams are being performed at once on one disk. + +"Front merges", a new request being merged at the front of an existing request, +are far less common than "back merges" due to the nature of most I/O patterns. +Front merges are handled by the binary trees in AS and deadline schedulers. + +iii. Plugging the queue to batch requests in anticipation of opportunities for + merge/sort optimizations + +Plugging is an approach that the current i/o scheduling algorithm resorts to so +that it collects up enough requests in the queue to be able to take +advantage of the sorting/merging logic in the elevator. If the +queue is empty when a request comes in, then it plugs the request queue +(sort of like plugging the bath tub of a vessel to get fluid to build up) +till it fills up with a few more requests, before starting to service +the requests. This provides an opportunity to merge/sort the requests before +passing them down to the device. There are various conditions when the queue is +unplugged (to open up the flow again), either through a scheduled task or +could be on demand. For example wait_on_buffer sets the unplugging going +through sync_buffer() running blk_run_address_space(mapping). Or the caller +can do it explicity through blk_unplug(bdev). So in the read case, +the queue gets explicitly unplugged as part of waiting for completion on that +buffer. + +Aside: + This is kind of controversial territory, as it's not clear if plugging is + always the right thing to do. Devices typically have their own queues, + and allowing a big queue to build up in software, while letting the device be + idle for a while may not always make sense. The trick is to handle the fine + balance between when to plug and when to open up. Also now that we have + multi-page bios being queued in one shot, we may not need to wait to merge + a big request from the broken up pieces coming by. + +4.4 I/O contexts +I/O contexts provide a dynamically allocated per process data area. They may +be used in I/O schedulers, and in the block layer (could be used for IO statis, +priorities for example). See *io_context in block/ll_rw_blk.c, and as-iosched.c +for an example of usage in an i/o scheduler. + + +5. Scalability related changes + +5.1 Granular Locking: io_request_lock replaced by a per-queue lock + +The global io_request_lock has been removed as of 2.5, to avoid +the scalability bottleneck it was causing, and has been replaced by more +granular locking. The request queue structure has a pointer to the +lock to be used for that queue. As a result, locking can now be +per-queue, with a provision for sharing a lock across queues if +necessary (e.g the scsi layer sets the queue lock pointers to the +corresponding adapter lock, which results in a per host locking +granularity). The locking semantics are the same, i.e. locking is +still imposed by the block layer, grabbing the lock before +request_fn execution which it means that lots of older drivers +should still be SMP safe. Drivers are free to drop the queue +lock themselves, if required. Drivers that explicitly used the +io_request_lock for serialization need to be modified accordingly. +Usually it's as easy as adding a global lock: + + static DEFINE_SPINLOCK(my_driver_lock); + +and passing the address to that lock to blk_init_queue(). + +5.2 64 bit sector numbers (sector_t prepares for 64 bit support) + +The sector number used in the bio structure has been changed to sector_t, +which could be defined as 64 bit in preparation for 64 bit sector support. + +6. Other Changes/Implications + +6.1 Partition re-mapping handled by the generic block layer + +In 2.5 some of the gendisk/partition related code has been reorganized. +Now the generic block layer performs partition-remapping early and thus +provides drivers with a sector number relative to whole device, rather than +having to take partition number into account in order to arrive at the true +sector number. The routine blk_partition_remap() is invoked by +generic_make_request even before invoking the queue specific make_request_fn, +so the i/o scheduler also gets to operate on whole disk sector numbers. This +should typically not require changes to block drivers, it just never gets +to invoke its own partition sector offset calculations since all bios +sent are offset from the beginning of the device. + + +7. A Few Tips on Migration of older drivers + +Old-style drivers that just use CURRENT and ignores clustered requests, +may not need much change. The generic layer will automatically handle +clustered requests, multi-page bios, etc for the driver. + +For a low performance driver or hardware that is PIO driven or just doesn't +support scatter-gather changes should be minimal too. + +The following are some points to keep in mind when converting old drivers +to bio. + +Drivers should use elv_next_request to pick up requests and are no longer +supposed to handle looping directly over the request list. +(struct request->queue has been removed) + +Now end_that_request_first takes an additional number_of_sectors argument. +It used to handle always just the first buffer_head in a request, now +it will loop and handle as many sectors (on a bio-segment granularity) +as specified. + +Now bh->b_end_io is replaced by bio->bi_end_io, but most of the time the +right thing to use is bio_endio(bio) instead. + +If the driver is dropping the io_request_lock from its request_fn strategy, +then it just needs to replace that with q->queue_lock instead. + +As described in Sec 1.1, drivers can set max sector size, max segment size +etc per queue now. Drivers that used to define their own merge functions i +to handle things like this can now just use the blk_queue_* functions at +blk_init_queue time. + +Drivers no longer have to map a {partition, sector offset} into the +correct absolute location anymore, this is done by the block layer, so +where a driver received a request ala this before: + + rq->rq_dev = mk_kdev(3, 5); /* /dev/hda5 */ + rq->sector = 0; /* first sector on hda5 */ + + it will now see + + rq->rq_dev = mk_kdev(3, 0); /* /dev/hda */ + rq->sector = 123128; /* offset from start of disk */ + +As mentioned, there is no virtual mapping of a bio. For DMA, this is +not a problem as the driver probably never will need a virtual mapping. +Instead it needs a bus mapping (dma_map_page for a single segment or +use dma_map_sg for scatter gather) to be able to ship it to the driver. For +PIO drivers (or drivers that need to revert to PIO transfer once in a +while (IDE for example)), where the CPU is doing the actual data +transfer a virtual mapping is needed. If the driver supports highmem I/O, +(Sec 1.1, (ii) ) it needs to use kmap_atomic or similar to temporarily map +a bio into the virtual address space. + + +8. Prior/Related/Impacted patches + +8.1. Earlier kiobuf patches (sct/axboe/chait/hch/mkp) +- orig kiobuf & raw i/o patches (now in 2.4 tree) +- direct kiobuf based i/o to devices (no intermediate bh's) +- page i/o using kiobuf +- kiobuf splitting for lvm (mkp) +- elevator support for kiobuf request merging (axboe) +8.2. Zero-copy networking (Dave Miller) +8.3. SGI XFS - pagebuf patches - use of kiobufs +8.4. Multi-page pioent patch for bio (Christoph Hellwig) +8.5. Direct i/o implementation (Andrea Arcangeli) since 2.4.10-pre11 +8.6. Async i/o implementation patch (Ben LaHaise) +8.7. EVMS layering design (IBM EVMS team) +8.8. Larger page cache size patch (Ben LaHaise) and + Large page size (Daniel Phillips) + => larger contiguous physical memory buffers +8.9. VM reservations patch (Ben LaHaise) +8.10. Write clustering patches ? (Marcelo/Quintela/Riel ?) +8.11. Block device in page cache patch (Andrea Archangeli) - now in 2.4.10+ +8.12. Multiple block-size transfers for faster raw i/o (Shailabh Nagar, + Badari) +8.13 Priority based i/o scheduler - prepatches (Arjan van de Ven) +8.14 IDE Taskfile i/o patch (Andre Hedrick) +8.15 Multi-page writeout and readahead patches (Andrew Morton) +8.16 Direct i/o patches for 2.5 using kvec and bio (Badari Pulavarthy) + +9. Other References: + +9.1 The Splice I/O Model - Larry McVoy (and subsequent discussions on lkml, +and Linus' comments - Jan 2001) +9.2 Discussions about kiobuf and bh design on lkml between sct, linus, alan +et al - Feb-March 2001 (many of the initial thoughts that led to bio were +brought up in this discussion thread) +9.3 Discussions on mempool on lkml - Dec 2001. + diff --git a/Documentation/block/biovecs.txt b/Documentation/block/biovecs.txt new file mode 100644 index 000000000..25689584e --- /dev/null +++ b/Documentation/block/biovecs.txt @@ -0,0 +1,119 @@ + +Immutable biovecs and biovec iterators: +======================================= + +Kent Overstreet <kmo@daterainc.com> + +As of 3.13, biovecs should never be modified after a bio has been submitted. +Instead, we have a new struct bvec_iter which represents a range of a biovec - +the iterator will be modified as the bio is completed, not the biovec. + +More specifically, old code that needed to partially complete a bio would +update bi_sector and bi_size, and advance bi_idx to the next biovec. If it +ended up partway through a biovec, it would increment bv_offset and decrement +bv_len by the number of bytes completed in that biovec. + +In the new scheme of things, everything that must be mutated in order to +partially complete a bio is segregated into struct bvec_iter: bi_sector, +bi_size and bi_idx have been moved there; and instead of modifying bv_offset +and bv_len, struct bvec_iter has bi_bvec_done, which represents the number of +bytes completed in the current bvec. + +There are a bunch of new helper macros for hiding the gory details - in +particular, presenting the illusion of partially completed biovecs so that +normal code doesn't have to deal with bi_bvec_done. + + * Driver code should no longer refer to biovecs directly; we now have + bio_iovec() and bio_iter_iovec() macros that return literal struct biovecs, + constructed from the raw biovecs but taking into account bi_bvec_done and + bi_size. + + bio_for_each_segment() has been updated to take a bvec_iter argument + instead of an integer (that corresponded to bi_idx); for a lot of code the + conversion just required changing the types of the arguments to + bio_for_each_segment(). + + * Advancing a bvec_iter is done with bio_advance_iter(); bio_advance() is a + wrapper around bio_advance_iter() that operates on bio->bi_iter, and also + advances the bio integrity's iter if present. + + There is a lower level advance function - bvec_iter_advance() - which takes + a pointer to a biovec, not a bio; this is used by the bio integrity code. + +What's all this get us? +======================= + +Having a real iterator, and making biovecs immutable, has a number of +advantages: + + * Before, iterating over bios was very awkward when you weren't processing + exactly one bvec at a time - for example, bio_copy_data() in fs/bio.c, + which copies the contents of one bio into another. Because the biovecs + wouldn't necessarily be the same size, the old code was tricky convoluted - + it had to walk two different bios at the same time, keeping both bi_idx and + and offset into the current biovec for each. + + The new code is much more straightforward - have a look. This sort of + pattern comes up in a lot of places; a lot of drivers were essentially open + coding bvec iterators before, and having common implementation considerably + simplifies a lot of code. + + * Before, any code that might need to use the biovec after the bio had been + completed (perhaps to copy the data somewhere else, or perhaps to resubmit + it somewhere else if there was an error) had to save the entire bvec array + - again, this was being done in a fair number of places. + + * Biovecs can be shared between multiple bios - a bvec iter can represent an + arbitrary range of an existing biovec, both starting and ending midway + through biovecs. This is what enables efficient splitting of arbitrary + bios. Note that this means we _only_ use bi_size to determine when we've + reached the end of a bio, not bi_vcnt - and the bio_iovec() macro takes + bi_size into account when constructing biovecs. + + * Splitting bios is now much simpler. The old bio_split() didn't even work on + bios with more than a single bvec! Now, we can efficiently split arbitrary + size bios - because the new bio can share the old bio's biovec. + + Care must be taken to ensure the biovec isn't freed while the split bio is + still using it, in case the original bio completes first, though. Using + bio_chain() when splitting bios helps with this. + + * Submitting partially completed bios is now perfectly fine - this comes up + occasionally in stacking block drivers and various code (e.g. md and + bcache) had some ugly workarounds for this. + + It used to be the case that submitting a partially completed bio would work + fine to _most_ devices, but since accessing the raw bvec array was the + norm, not all drivers would respect bi_idx and those would break. Now, + since all drivers _must_ go through the bvec iterator - and have been + audited to make sure they are - submitting partially completed bios is + perfectly fine. + +Other implications: +=================== + + * Almost all usage of bi_idx is now incorrect and has been removed; instead, + where previously you would have used bi_idx you'd now use a bvec_iter, + probably passing it to one of the helper macros. + + I.e. instead of using bio_iovec_idx() (or bio->bi_iovec[bio->bi_idx]), you + now use bio_iter_iovec(), which takes a bvec_iter and returns a + literal struct bio_vec - constructed on the fly from the raw biovec but + taking into account bi_bvec_done (and bi_size). + + * bi_vcnt can't be trusted or relied upon by driver code - i.e. anything that + doesn't actually own the bio. The reason is twofold: firstly, it's not + actually needed for iterating over the bio anymore - we only use bi_size. + Secondly, when cloning a bio and reusing (a portion of) the original bio's + biovec, in order to calculate bi_vcnt for the new bio we'd have to iterate + over all the biovecs in the new bio - which is silly as it's not needed. + + So, don't use bi_vcnt anymore. + + * The current interface allows the block layer to split bios as needed, so we + could eliminate a lot of complexity particularly in stacked drivers. Code + that creates bios can then create whatever size bios are convenient, and + more importantly stacked drivers don't have to deal with both their own bio + size limitations and the limitations of the underlying devices. Thus + there's no need to define ->merge_bvec_fn() callbacks for individual block + drivers. diff --git a/Documentation/block/capability.txt b/Documentation/block/capability.txt new file mode 100644 index 000000000..2f1729424 --- /dev/null +++ b/Documentation/block/capability.txt @@ -0,0 +1,15 @@ +Generic Block Device Capability +=============================================================================== +This file documents the sysfs file block/<disk>/capability + +capability is a hex word indicating which capabilities a specific disk +supports. For more information on bits not listed here, see +include/linux/genhd.h + +Capability Value +------------------------------------------------------------------------------- +GENHD_FL_MEDIA_CHANGE_NOTIFY 4 + When this bit is set, the disk supports Asynchronous Notification + of media change events. These events will be broadcast to user + space via kernel uevent. + diff --git a/Documentation/block/cfq-iosched.txt b/Documentation/block/cfq-iosched.txt new file mode 100644 index 000000000..895bd3813 --- /dev/null +++ b/Documentation/block/cfq-iosched.txt @@ -0,0 +1,291 @@ +CFQ (Complete Fairness Queueing) +=============================== + +The main aim of CFQ scheduler is to provide a fair allocation of the disk +I/O bandwidth for all the processes which requests an I/O operation. + +CFQ maintains the per process queue for the processes which request I/O +operation(synchronous requests). In case of asynchronous requests, all the +requests from all the processes are batched together according to their +process's I/O priority. + +CFQ ioscheduler tunables +======================== + +slice_idle +---------- +This specifies how long CFQ should idle for next request on certain cfq queues +(for sequential workloads) and service trees (for random workloads) before +queue is expired and CFQ selects next queue to dispatch from. + +By default slice_idle is a non-zero value. That means by default we idle on +queues/service trees. This can be very helpful on highly seeky media like +single spindle SATA/SAS disks where we can cut down on overall number of +seeks and see improved throughput. + +Setting slice_idle to 0 will remove all the idling on queues/service tree +level and one should see an overall improved throughput on faster storage +devices like multiple SATA/SAS disks in hardware RAID configuration. The down +side is that isolation provided from WRITES also goes down and notion of +IO priority becomes weaker. + +So depending on storage and workload, it might be useful to set slice_idle=0. +In general I think for SATA/SAS disks and software RAID of SATA/SAS disks +keeping slice_idle enabled should be useful. For any configurations where +there are multiple spindles behind single LUN (Host based hardware RAID +controller or for storage arrays), setting slice_idle=0 might end up in better +throughput and acceptable latencies. + +back_seek_max +------------- +This specifies, given in Kbytes, the maximum "distance" for backward seeking. +The distance is the amount of space from the current head location to the +sectors that are backward in terms of distance. + +This parameter allows the scheduler to anticipate requests in the "backward" +direction and consider them as being the "next" if they are within this +distance from the current head location. + +back_seek_penalty +----------------- +This parameter is used to compute the cost of backward seeking. If the +backward distance of request is just 1/back_seek_penalty from a "front" +request, then the seeking cost of two requests is considered equivalent. + +So scheduler will not bias toward one or the other request (otherwise scheduler +will bias toward front request). Default value of back_seek_penalty is 2. + +fifo_expire_async +----------------- +This parameter is used to set the timeout of asynchronous requests. Default +value of this is 248ms. + +fifo_expire_sync +---------------- +This parameter is used to set the timeout of synchronous requests. Default +value of this is 124ms. In case to favor synchronous requests over asynchronous +one, this value should be decreased relative to fifo_expire_async. + +group_idle +----------- +This parameter forces idling at the CFQ group level instead of CFQ +queue level. This was introduced after a bottleneck was observed +in higher end storage due to idle on sequential queue and allow dispatch +from a single queue. The idea with this parameter is that it can be run with +slice_idle=0 and group_idle=8, so that idling does not happen on individual +queues in the group but happens overall on the group and thus still keeps the +IO controller working. +Not idling on individual queues in the group will dispatch requests from +multiple queues in the group at the same time and achieve higher throughput +on higher end storage. + +Default value for this parameter is 8ms. + +low_latency +----------- +This parameter is used to enable/disable the low latency mode of the CFQ +scheduler. If enabled, CFQ tries to recompute the slice time for each process +based on the target_latency set for the system. This favors fairness over +throughput. Disabling low latency (setting it to 0) ignores target latency, +allowing each process in the system to get a full time slice. + +By default low latency mode is enabled. + +target_latency +-------------- +This parameter is used to calculate the time slice for a process if cfq's +latency mode is enabled. It will ensure that sync requests have an estimated +latency. But if sequential workload is higher(e.g. sequential read), +then to meet the latency constraints, throughput may decrease because of less +time for each process to issue I/O request before the cfq queue is switched. + +Though this can be overcome by disabling the latency_mode, it may increase +the read latency for some applications. This parameter allows for changing +target_latency through the sysfs interface which can provide the balanced +throughput and read latency. + +Default value for target_latency is 300ms. + +slice_async +----------- +This parameter is same as of slice_sync but for asynchronous queue. The +default value is 40ms. + +slice_async_rq +-------------- +This parameter is used to limit the dispatching of asynchronous request to +device request queue in queue's slice time. The maximum number of request that +are allowed to be dispatched also depends upon the io priority. Default value +for this is 2. + +slice_sync +---------- +When a queue is selected for execution, the queues IO requests are only +executed for a certain amount of time(time_slice) before switching to another +queue. This parameter is used to calculate the time slice of synchronous +queue. + +time_slice is computed using the below equation:- +time_slice = slice_sync + (slice_sync/5 * (4 - prio)). To increase the +time_slice of synchronous queue, increase the value of slice_sync. Default +value is 100ms. + +quantum +------- +This specifies the number of request dispatched to the device queue. In a +queue's time slice, a request will not be dispatched if the number of request +in the device exceeds this parameter. This parameter is used for synchronous +request. + +In case of storage with several disk, this setting can limit the parallel +processing of request. Therefore, increasing the value can improve the +performance although this can cause the latency of some I/O to increase due +to more number of requests. + +CFQ Group scheduling +==================== + +CFQ supports blkio cgroup and has "blkio." prefixed files in each +blkio cgroup directory. It is weight-based and there are four knobs +for configuration - weight[_device] and leaf_weight[_device]. +Internal cgroup nodes (the ones with children) can also have tasks in +them, so the former two configure how much proportion the cgroup as a +whole is entitled to at its parent's level while the latter two +configure how much proportion the tasks in the cgroup have compared to +its direct children. + +Another way to think about it is assuming that each internal node has +an implicit leaf child node which hosts all the tasks whose weight is +configured by leaf_weight[_device]. Let's assume a blkio hierarchy +composed of five cgroups - root, A, B, AA and AB - with the following +weights where the names represent the hierarchy. + + weight leaf_weight + root : 125 125 + A : 500 750 + B : 250 500 + AA : 500 500 + AB : 1000 500 + +root never has a parent making its weight is meaningless. For backward +compatibility, weight is always kept in sync with leaf_weight. B, AA +and AB have no child and thus its tasks have no children cgroup to +compete with. They always get 100% of what the cgroup won at the +parent level. Considering only the weights which matter, the hierarchy +looks like the following. + + root + / | \ + A B leaf + 500 250 125 + / | \ + AA AB leaf + 500 1000 750 + +If all cgroups have active IOs and competing with each other, disk +time will be distributed like the following. + +Distribution below root. The total active weight at this level is +A:500 + B:250 + C:125 = 875. + + root-leaf : 125 / 875 =~ 14% + A : 500 / 875 =~ 57% + B(-leaf) : 250 / 875 =~ 28% + +A has children and further distributes its 57% among the children and +the implicit leaf node. The total active weight at this level is +AA:500 + AB:1000 + A-leaf:750 = 2250. + + A-leaf : ( 750 / 2250) * A =~ 19% + AA(-leaf) : ( 500 / 2250) * A =~ 12% + AB(-leaf) : (1000 / 2250) * A =~ 25% + +CFQ IOPS Mode for group scheduling +=================================== +Basic CFQ design is to provide priority based time slices. Higher priority +process gets bigger time slice and lower priority process gets smaller time +slice. Measuring time becomes harder if storage is fast and supports NCQ and +it would be better to dispatch multiple requests from multiple cfq queues in +request queue at a time. In such scenario, it is not possible to measure time +consumed by single queue accurately. + +What is possible though is to measure number of requests dispatched from a +single queue and also allow dispatch from multiple cfq queue at the same time. +This effectively becomes the fairness in terms of IOPS (IO operations per +second). + +If one sets slice_idle=0 and if storage supports NCQ, CFQ internally switches +to IOPS mode and starts providing fairness in terms of number of requests +dispatched. Note that this mode switching takes effect only for group +scheduling. For non-cgroup users nothing should change. + +CFQ IO scheduler Idling Theory +=============================== +Idling on a queue is primarily about waiting for the next request to come +on same queue after completion of a request. In this process CFQ will not +dispatch requests from other cfq queues even if requests are pending there. + +The rationale behind idling is that it can cut down on number of seeks +on rotational media. For example, if a process is doing dependent +sequential reads (next read will come on only after completion of previous +one), then not dispatching request from other queue should help as we +did not move the disk head and kept on dispatching sequential IO from +one queue. + +CFQ has following service trees and various queues are put on these trees. + + sync-idle sync-noidle async + +All cfq queues doing synchronous sequential IO go on to sync-idle tree. +On this tree we idle on each queue individually. + +All synchronous non-sequential queues go on sync-noidle tree. Also any +synchronous write request which is not marked with REQ_IDLE goes on this +service tree. On this tree we do not idle on individual queues instead idle +on the whole group of queues or the tree. So if there are 4 queues waiting +for IO to dispatch we will idle only once last queue has dispatched the IO +and there is no more IO on this service tree. + +All async writes go on async service tree. There is no idling on async +queues. + +CFQ has some optimizations for SSDs and if it detects a non-rotational +media which can support higher queue depth (multiple requests at in +flight at a time), then it cuts down on idling of individual queues and +all the queues move to sync-noidle tree and only tree idle remains. This +tree idling provides isolation with buffered write queues on async tree. + +FAQ +=== +Q1. Why to idle at all on queues not marked with REQ_IDLE. + +A1. We only do tree idle (all queues on sync-noidle tree) on queues not marked + with REQ_IDLE. This helps in providing isolation with all the sync-idle + queues. Otherwise in presence of many sequential readers, other + synchronous IO might not get fair share of disk. + + For example, if there are 10 sequential readers doing IO and they get + 100ms each. If a !REQ_IDLE request comes in, it will be scheduled + roughly after 1 second. If after completion of !REQ_IDLE request we + do not idle, and after a couple of milli seconds a another !REQ_IDLE + request comes in, again it will be scheduled after 1second. Repeat it + and notice how a workload can lose its disk share and suffer due to + multiple sequential readers. + + fsync can generate dependent IO where bunch of data is written in the + context of fsync, and later some journaling data is written. Journaling + data comes in only after fsync has finished its IO (atleast for ext4 + that seemed to be the case). Now if one decides not to idle on fsync + thread due to !REQ_IDLE, then next journaling write will not get + scheduled for another second. A process doing small fsync, will suffer + badly in presence of multiple sequential readers. + + Hence doing tree idling on threads using !REQ_IDLE flag on requests + provides isolation from multiple sequential readers and at the same + time we do not idle on individual threads. + +Q2. When to specify REQ_IDLE +A2. I would think whenever one is doing synchronous write and expecting + more writes to be dispatched from same context soon, should be able + to specify REQ_IDLE on writes and that probably should work well for + most of the cases. diff --git a/Documentation/block/cmdline-partition.txt b/Documentation/block/cmdline-partition.txt new file mode 100644 index 000000000..760a3f7c3 --- /dev/null +++ b/Documentation/block/cmdline-partition.txt @@ -0,0 +1,46 @@ +Embedded device command line partition parsing +===================================================================== + +The "blkdevparts" command line option adds support for reading the +block device partition table from the kernel command line. + +It is typically used for fixed block (eMMC) embedded devices. +It has no MBR, so saves storage space. Bootloader can be easily accessed +by absolute address of data on the block device. +Users can easily change the partition. + +The format for the command line is just like mtdparts: + +blkdevparts=<blkdev-def>[;<blkdev-def>] + <blkdev-def> := <blkdev-id>:<partdef>[,<partdef>] + <partdef> := <size>[@<offset>](part-name) + +<blkdev-id> + block device disk name. Embedded device uses fixed block device. + Its disk name is also fixed, such as: mmcblk0, mmcblk1, mmcblk0boot0. + +<size> + partition size, in bytes, such as: 512, 1m, 1G. + size may contain an optional suffix of (upper or lower case): + K, M, G, T, P, E. + "-" is used to denote all remaining space. + +<offset> + partition start address, in bytes. + offset may contain an optional suffix of (upper or lower case): + K, M, G, T, P, E. + +(part-name) + partition name. Kernel sends uevent with "PARTNAME". Application can + create a link to block device partition with the name "PARTNAME". + User space application can access partition by partition name. + +Example: + eMMC disk names are "mmcblk0" and "mmcblk0boot0". + + bootargs: + 'blkdevparts=mmcblk0:1G(data0),1G(data1),-;mmcblk0boot0:1m(boot),-(kernel)' + + dmesg: + mmcblk0: p1(data0) p2(data1) p3() + mmcblk0boot0: p1(boot) p2(kernel) diff --git a/Documentation/block/data-integrity.txt b/Documentation/block/data-integrity.txt new file mode 100644 index 000000000..934c44ea0 --- /dev/null +++ b/Documentation/block/data-integrity.txt @@ -0,0 +1,281 @@ +---------------------------------------------------------------------- +1. INTRODUCTION + +Modern filesystems feature checksumming of data and metadata to +protect against data corruption. However, the detection of the +corruption is done at read time which could potentially be months +after the data was written. At that point the original data that the +application tried to write is most likely lost. + +The solution is to ensure that the disk is actually storing what the +application meant it to. Recent additions to both the SCSI family +protocols (SBC Data Integrity Field, SCC protection proposal) as well +as SATA/T13 (External Path Protection) try to remedy this by adding +support for appending integrity metadata to an I/O. The integrity +metadata (or protection information in SCSI terminology) includes a +checksum for each sector as well as an incrementing counter that +ensures the individual sectors are written in the right order. And +for some protection schemes also that the I/O is written to the right +place on disk. + +Current storage controllers and devices implement various protective +measures, for instance checksumming and scrubbing. But these +technologies are working in their own isolated domains or at best +between adjacent nodes in the I/O path. The interesting thing about +DIF and the other integrity extensions is that the protection format +is well defined and every node in the I/O path can verify the +integrity of the I/O and reject it if corruption is detected. This +allows not only corruption prevention but also isolation of the point +of failure. + +---------------------------------------------------------------------- +2. THE DATA INTEGRITY EXTENSIONS + +As written, the protocol extensions only protect the path between +controller and storage device. However, many controllers actually +allow the operating system to interact with the integrity metadata +(IMD). We have been working with several FC/SAS HBA vendors to enable +the protection information to be transferred to and from their +controllers. + +The SCSI Data Integrity Field works by appending 8 bytes of protection +information to each sector. The data + integrity metadata is stored +in 520 byte sectors on disk. Data + IMD are interleaved when +transferred between the controller and target. The T13 proposal is +similar. + +Because it is highly inconvenient for operating systems to deal with +520 (and 4104) byte sectors, we approached several HBA vendors and +encouraged them to allow separation of the data and integrity metadata +scatter-gather lists. + +The controller will interleave the buffers on write and split them on +read. This means that Linux can DMA the data buffers to and from +host memory without changes to the page cache. + +Also, the 16-bit CRC checksum mandated by both the SCSI and SATA specs +is somewhat heavy to compute in software. Benchmarks found that +calculating this checksum had a significant impact on system +performance for a number of workloads. Some controllers allow a +lighter-weight checksum to be used when interfacing with the operating +system. Emulex, for instance, supports the TCP/IP checksum instead. +The IP checksum received from the OS is converted to the 16-bit CRC +when writing and vice versa. This allows the integrity metadata to be +generated by Linux or the application at very low cost (comparable to +software RAID5). + +The IP checksum is weaker than the CRC in terms of detecting bit +errors. However, the strength is really in the separation of the data +buffers and the integrity metadata. These two distinct buffers must +match up for an I/O to complete. + +The separation of the data and integrity metadata buffers as well as +the choice in checksums is referred to as the Data Integrity +Extensions. As these extensions are outside the scope of the protocol +bodies (T10, T13), Oracle and its partners are trying to standardize +them within the Storage Networking Industry Association. + +---------------------------------------------------------------------- +3. KERNEL CHANGES + +The data integrity framework in Linux enables protection information +to be pinned to I/Os and sent to/received from controllers that +support it. + +The advantage to the integrity extensions in SCSI and SATA is that +they enable us to protect the entire path from application to storage +device. However, at the same time this is also the biggest +disadvantage. It means that the protection information must be in a +format that can be understood by the disk. + +Generally Linux/POSIX applications are agnostic to the intricacies of +the storage devices they are accessing. The virtual filesystem switch +and the block layer make things like hardware sector size and +transport protocols completely transparent to the application. + +However, this level of detail is required when preparing the +protection information to send to a disk. Consequently, the very +concept of an end-to-end protection scheme is a layering violation. +It is completely unreasonable for an application to be aware whether +it is accessing a SCSI or SATA disk. + +The data integrity support implemented in Linux attempts to hide this +from the application. As far as the application (and to some extent +the kernel) is concerned, the integrity metadata is opaque information +that's attached to the I/O. + +The current implementation allows the block layer to automatically +generate the protection information for any I/O. Eventually the +intent is to move the integrity metadata calculation to userspace for +user data. Metadata and other I/O that originates within the kernel +will still use the automatic generation interface. + +Some storage devices allow each hardware sector to be tagged with a +16-bit value. The owner of this tag space is the owner of the block +device. I.e. the filesystem in most cases. The filesystem can use +this extra space to tag sectors as they see fit. Because the tag +space is limited, the block interface allows tagging bigger chunks by +way of interleaving. This way, 8*16 bits of information can be +attached to a typical 4KB filesystem block. + +This also means that applications such as fsck and mkfs will need +access to manipulate the tags from user space. A passthrough +interface for this is being worked on. + + +---------------------------------------------------------------------- +4. BLOCK LAYER IMPLEMENTATION DETAILS + +4.1 BIO + +The data integrity patches add a new field to struct bio when +CONFIG_BLK_DEV_INTEGRITY is enabled. bio_integrity(bio) returns a +pointer to a struct bip which contains the bio integrity payload. +Essentially a bip is a trimmed down struct bio which holds a bio_vec +containing the integrity metadata and the required housekeeping +information (bvec pool, vector count, etc.) + +A kernel subsystem can enable data integrity protection on a bio by +calling bio_integrity_alloc(bio). This will allocate and attach the +bip to the bio. + +Individual pages containing integrity metadata can subsequently be +attached using bio_integrity_add_page(). + +bio_free() will automatically free the bip. + + +4.2 BLOCK DEVICE + +Because the format of the protection data is tied to the physical +disk, each block device has been extended with a block integrity +profile (struct blk_integrity). This optional profile is registered +with the block layer using blk_integrity_register(). + +The profile contains callback functions for generating and verifying +the protection data, as well as getting and setting application tags. +The profile also contains a few constants to aid in completing, +merging and splitting the integrity metadata. + +Layered block devices will need to pick a profile that's appropriate +for all subdevices. blk_integrity_compare() can help with that. DM +and MD linear, RAID0 and RAID1 are currently supported. RAID4/5/6 +will require extra work due to the application tag. + + +---------------------------------------------------------------------- +5.0 BLOCK LAYER INTEGRITY API + +5.1 NORMAL FILESYSTEM + + The normal filesystem is unaware that the underlying block device + is capable of sending/receiving integrity metadata. The IMD will + be automatically generated by the block layer at submit_bio() time + in case of a WRITE. A READ request will cause the I/O integrity + to be verified upon completion. + + IMD generation and verification can be toggled using the + + /sys/block/<bdev>/integrity/write_generate + + and + + /sys/block/<bdev>/integrity/read_verify + + flags. + + +5.2 INTEGRITY-AWARE FILESYSTEM + + A filesystem that is integrity-aware can prepare I/Os with IMD + attached. It can also use the application tag space if this is + supported by the block device. + + + bool bio_integrity_prep(bio); + + To generate IMD for WRITE and to set up buffers for READ, the + filesystem must call bio_integrity_prep(bio). + + Prior to calling this function, the bio data direction and start + sector must be set, and the bio should have all data pages + added. It is up to the caller to ensure that the bio does not + change while I/O is in progress. + Complete bio with error if prepare failed for some reson. + + +5.3 PASSING EXISTING INTEGRITY METADATA + + Filesystems that either generate their own integrity metadata or + are capable of transferring IMD from user space can use the + following calls: + + + struct bip * bio_integrity_alloc(bio, gfp_mask, nr_pages); + + Allocates the bio integrity payload and hangs it off of the bio. + nr_pages indicate how many pages of protection data need to be + stored in the integrity bio_vec list (similar to bio_alloc()). + + The integrity payload will be freed at bio_free() time. + + + int bio_integrity_add_page(bio, page, len, offset); + + Attaches a page containing integrity metadata to an existing + bio. The bio must have an existing bip, + i.e. bio_integrity_alloc() must have been called. For a WRITE, + the integrity metadata in the pages must be in a format + understood by the target device with the notable exception that + the sector numbers will be remapped as the request traverses the + I/O stack. This implies that the pages added using this call + will be modified during I/O! The first reference tag in the + integrity metadata must have a value of bip->bip_sector. + + Pages can be added using bio_integrity_add_page() as long as + there is room in the bip bio_vec array (nr_pages). + + Upon completion of a READ operation, the attached pages will + contain the integrity metadata received from the storage device. + It is up to the receiver to process them and verify data + integrity upon completion. + + +5.4 REGISTERING A BLOCK DEVICE AS CAPABLE OF EXCHANGING INTEGRITY + METADATA + + To enable integrity exchange on a block device the gendisk must be + registered as capable: + + int blk_integrity_register(gendisk, blk_integrity); + + The blk_integrity struct is a template and should contain the + following: + + static struct blk_integrity my_profile = { + .name = "STANDARDSBODY-TYPE-VARIANT-CSUM", + .generate_fn = my_generate_fn, + .verify_fn = my_verify_fn, + .tuple_size = sizeof(struct my_tuple_size), + .tag_size = <tag bytes per hw sector>, + }; + + 'name' is a text string which will be visible in sysfs. This is + part of the userland API so chose it carefully and never change + it. The format is standards body-type-variant. + E.g. T10-DIF-TYPE1-IP or T13-EPP-0-CRC. + + 'generate_fn' generates appropriate integrity metadata (for WRITE). + + 'verify_fn' verifies that the data buffer matches the integrity + metadata. + + 'tuple_size' must be set to match the size of the integrity + metadata per sector. I.e. 8 for DIF and EPP. + + 'tag_size' must be set to identify how many bytes of tag space + are available per hardware sector. For DIF this is either 2 or + 0 depending on the value of the Control Mode Page ATO bit. + +---------------------------------------------------------------------- +2007-12-24 Martin K. Petersen <martin.petersen@oracle.com> diff --git a/Documentation/block/deadline-iosched.txt b/Documentation/block/deadline-iosched.txt new file mode 100644 index 000000000..2d82c8032 --- /dev/null +++ b/Documentation/block/deadline-iosched.txt @@ -0,0 +1,75 @@ +Deadline IO scheduler tunables +============================== + +This little file attempts to document how the deadline io scheduler works. +In particular, it will clarify the meaning of the exposed tunables that may be +of interest to power users. + +Selecting IO schedulers +----------------------- +Refer to Documentation/block/switching-sched.txt for information on +selecting an io scheduler on a per-device basis. + + +******************************************************************************** + + +read_expire (in ms) +----------- + +The goal of the deadline io scheduler is to attempt to guarantee a start +service time for a request. As we focus mainly on read latencies, this is +tunable. When a read request first enters the io scheduler, it is assigned +a deadline that is the current time + the read_expire value in units of +milliseconds. + + +write_expire (in ms) +----------- + +Similar to read_expire mentioned above, but for writes. + + +fifo_batch (number of requests) +---------- + +Requests are grouped into ``batches'' of a particular data direction (read or +write) which are serviced in increasing sector order. To limit extra seeking, +deadline expiries are only checked between batches. fifo_batch controls the +maximum number of requests per batch. + +This parameter tunes the balance between per-request latency and aggregate +throughput. When low latency is the primary concern, smaller is better (where +a value of 1 yields first-come first-served behaviour). Increasing fifo_batch +generally improves throughput, at the cost of latency variation. + + +writes_starved (number of dispatches) +-------------- + +When we have to move requests from the io scheduler queue to the block +device dispatch queue, we always give a preference to reads. However, we +don't want to starve writes indefinitely either. So writes_starved controls +how many times we give preference to reads over writes. When that has been +done writes_starved number of times, we dispatch some writes based on the +same criteria as reads. + + +front_merges (bool) +------------ + +Sometimes it happens that a request enters the io scheduler that is contiguous +with a request that is already on the queue. Either it fits in the back of that +request, or it fits at the front. That is called either a back merge candidate +or a front merge candidate. Due to the way files are typically laid out, +back merges are much more common than front merges. For some work loads, you +may even know that it is a waste of time to spend any time attempting to +front merge requests. Setting front_merges to 0 disables this functionality. +Front merges may still occur due to the cached last_merge hint, but since +that comes at basically 0 cost we leave that on. We simply disable the +rbtree front sector lookup when the io scheduler merge function is called. + + +Nov 11 2002, Jens Axboe <jens.axboe@oracle.com> + + diff --git a/Documentation/block/ioprio.txt b/Documentation/block/ioprio.txt new file mode 100644 index 000000000..8ed8c5938 --- /dev/null +++ b/Documentation/block/ioprio.txt @@ -0,0 +1,183 @@ +Block io priorities +=================== + + +Intro +----- + +With the introduction of cfq v3 (aka cfq-ts or time sliced cfq), basic io +priorities are supported for reads on files. This enables users to io nice +processes or process groups, similar to what has been possible with cpu +scheduling for ages. This document mainly details the current possibilities +with cfq; other io schedulers do not support io priorities thus far. + +Scheduling classes +------------------ + +CFQ implements three generic scheduling classes that determine how io is +served for a process. + +IOPRIO_CLASS_RT: This is the realtime io class. This scheduling class is given +higher priority than any other in the system, processes from this class are +given first access to the disk every time. Thus it needs to be used with some +care, one io RT process can starve the entire system. Within the RT class, +there are 8 levels of class data that determine exactly how much time this +process needs the disk for on each service. In the future this might change +to be more directly mappable to performance, by passing in a wanted data +rate instead. + +IOPRIO_CLASS_BE: This is the best-effort scheduling class, which is the default +for any process that hasn't set a specific io priority. The class data +determines how much io bandwidth the process will get, it's directly mappable +to the cpu nice levels just more coarsely implemented. 0 is the highest +BE prio level, 7 is the lowest. The mapping between cpu nice level and io +nice level is determined as: io_nice = (cpu_nice + 20) / 5. + +IOPRIO_CLASS_IDLE: This is the idle scheduling class, processes running at this +level only get io time when no one else needs the disk. The idle class has no +class data, since it doesn't really apply here. + +Tools +----- + +See below for a sample ionice tool. Usage: + +# ionice -c<class> -n<level> -p<pid> + +If pid isn't given, the current process is assumed. IO priority settings +are inherited on fork, so you can use ionice to start the process at a given +level: + +# ionice -c2 -n0 /bin/ls + +will run ls at the best-effort scheduling class at the highest priority. +For a running process, you can give the pid instead: + +# ionice -c1 -n2 -p100 + +will change pid 100 to run at the realtime scheduling class, at priority 2. + +---> snip ionice.c tool <--- + +#include <stdio.h> +#include <stdlib.h> +#include <errno.h> +#include <getopt.h> +#include <unistd.h> +#include <sys/ptrace.h> +#include <asm/unistd.h> + +extern int sys_ioprio_set(int, int, int); +extern int sys_ioprio_get(int, int); + +#if defined(__i386__) +#define __NR_ioprio_set 289 +#define __NR_ioprio_get 290 +#elif defined(__ppc__) +#define __NR_ioprio_set 273 +#define __NR_ioprio_get 274 +#elif defined(__x86_64__) +#define __NR_ioprio_set 251 +#define __NR_ioprio_get 252 +#elif defined(__ia64__) +#define __NR_ioprio_set 1274 +#define __NR_ioprio_get 1275 +#else +#error "Unsupported arch" +#endif + +static inline int ioprio_set(int which, int who, int ioprio) +{ + return syscall(__NR_ioprio_set, which, who, ioprio); +} + +static inline int ioprio_get(int which, int who) +{ + return syscall(__NR_ioprio_get, which, who); +} + +enum { + IOPRIO_CLASS_NONE, + IOPRIO_CLASS_RT, + IOPRIO_CLASS_BE, + IOPRIO_CLASS_IDLE, +}; + +enum { + IOPRIO_WHO_PROCESS = 1, + IOPRIO_WHO_PGRP, + IOPRIO_WHO_USER, +}; + +#define IOPRIO_CLASS_SHIFT 13 + +const char *to_prio[] = { "none", "realtime", "best-effort", "idle", }; + +int main(int argc, char *argv[]) +{ + int ioprio = 4, set = 0, ioprio_class = IOPRIO_CLASS_BE; + int c, pid = 0; + + while ((c = getopt(argc, argv, "+n:c:p:")) != EOF) { + switch (c) { + case 'n': + ioprio = strtol(optarg, NULL, 10); + set = 1; + break; + case 'c': + ioprio_class = strtol(optarg, NULL, 10); + set = 1; + break; + case 'p': + pid = strtol(optarg, NULL, 10); + break; + } + } + + switch (ioprio_class) { + case IOPRIO_CLASS_NONE: + ioprio_class = IOPRIO_CLASS_BE; + break; + case IOPRIO_CLASS_RT: + case IOPRIO_CLASS_BE: + break; + case IOPRIO_CLASS_IDLE: + ioprio = 7; + break; + default: + printf("bad prio class %d\n", ioprio_class); + return 1; + } + + if (!set) { + if (!pid && argv[optind]) + pid = strtol(argv[optind], NULL, 10); + + ioprio = ioprio_get(IOPRIO_WHO_PROCESS, pid); + + printf("pid=%d, %d\n", pid, ioprio); + + if (ioprio == -1) + perror("ioprio_get"); + else { + ioprio_class = ioprio >> IOPRIO_CLASS_SHIFT; + ioprio = ioprio & 0xff; + printf("%s: prio %d\n", to_prio[ioprio_class], ioprio); + } + } else { + if (ioprio_set(IOPRIO_WHO_PROCESS, pid, ioprio | ioprio_class << IOPRIO_CLASS_SHIFT) == -1) { + perror("ioprio_set"); + return 1; + } + + if (argv[optind]) + execvp(argv[optind], &argv[optind]); + } + + return 0; +} + +---> snip ionice.c tool <--- + + +March 11 2005, Jens Axboe <jens.axboe@oracle.com> diff --git a/Documentation/block/kyber-iosched.txt b/Documentation/block/kyber-iosched.txt new file mode 100644 index 000000000..e94feacd7 --- /dev/null +++ b/Documentation/block/kyber-iosched.txt @@ -0,0 +1,14 @@ +Kyber I/O scheduler tunables +=========================== + +The only two tunables for the Kyber scheduler are the target latencies for +reads and synchronous writes. Kyber will throttle requests in order to meet +these target latencies. + +read_lat_nsec +------------- +Target latency for reads (in nanoseconds). + +write_lat_nsec +-------------- +Target latency for synchronous writes (in nanoseconds). diff --git a/Documentation/block/null_blk.txt b/Documentation/block/null_blk.txt new file mode 100644 index 000000000..ea2dafe49 --- /dev/null +++ b/Documentation/block/null_blk.txt @@ -0,0 +1,94 @@ +Null block device driver +================================================================================ + +I. Overview + +The null block device (/dev/nullb*) is used for benchmarking the various +block-layer implementations. It emulates a block device of X gigabytes in size. +The following instances are possible: + + Single-queue block-layer + - Request-based. + - Single submission queue per device. + - Implements IO scheduling algorithms (CFQ, Deadline, noop). + Multi-queue block-layer + - Request-based. + - Configurable submission queues per device. + No block-layer (Known as bio-based) + - Bio-based. IO requests are submitted directly to the device driver. + - Directly accepts bio data structure and returns them. + +All of them have a completion queue for each core in the system. + +II. Module parameters applicable for all instances: + +queue_mode=[0-2]: Default: 2-Multi-queue + Selects which block-layer the module should instantiate with. + + 0: Bio-based. + 1: Single-queue. + 2: Multi-queue. + +home_node=[0--nr_nodes]: Default: NUMA_NO_NODE + Selects what CPU node the data structures are allocated from. + +gb=[Size in GB]: Default: 250GB + The size of the device reported to the system. + +bs=[Block size (in bytes)]: Default: 512 bytes + The block size reported to the system. + +nr_devices=[Number of devices]: Default: 1 + Number of block devices instantiated. They are instantiated as /dev/nullb0, + etc. + +irqmode=[0-2]: Default: 1-Soft-irq + The completion mode used for completing IOs to the block-layer. + + 0: None. + 1: Soft-irq. Uses IPI to complete IOs across CPU nodes. Simulates the overhead + when IOs are issued from another CPU node than the home the device is + connected to. + 2: Timer: Waits a specific period (completion_nsec) for each IO before + completion. + +completion_nsec=[ns]: Default: 10,000ns + Combined with irqmode=2 (timer). The time each completion event must wait. + +submit_queues=[1..nr_cpus]: + The number of submission queues attached to the device driver. If unset, it + defaults to 1. For multi-queue, it is ignored when use_per_node_hctx module + parameter is 1. + +hw_queue_depth=[0..qdepth]: Default: 64 + The hardware queue depth of the device. + +III: Multi-queue specific parameters + +use_per_node_hctx=[0/1]: Default: 0 + 0: The number of submit queues are set to the value of the submit_queues + parameter. + 1: The multi-queue block layer is instantiated with a hardware dispatch + queue for each CPU node in the system. + +no_sched=[0/1]: Default: 0 + 0: nullb* use default blk-mq io scheduler. + 1: nullb* doesn't use io scheduler. + +blocking=[0/1]: Default: 0 + 0: Register as a non-blocking blk-mq driver device. + 1: Register as a blocking blk-mq driver device, null_blk will set + the BLK_MQ_F_BLOCKING flag, indicating that it sometimes/always + needs to block in its ->queue_rq() function. + +shared_tags=[0/1]: Default: 0 + 0: Tag set is not shared. + 1: Tag set shared between devices for blk-mq. Only makes sense with + nr_devices > 1, otherwise there's no tag set to share. + +zoned=[0/1]: Default: 0 + 0: Block device is exposed as a random-access block device. + 1: Block device is exposed as a host-managed zoned block device. + +zone_size=[MB]: Default: 256 + Per zone size when exposed as a zoned block device. Must be a power of two. diff --git a/Documentation/block/pr.txt b/Documentation/block/pr.txt new file mode 100644 index 000000000..ac9b8e70e --- /dev/null +++ b/Documentation/block/pr.txt @@ -0,0 +1,119 @@ + +Block layer support for Persistent Reservations +=============================================== + +The Linux kernel supports a user space interface for simplified +Persistent Reservations which map to block devices that support +these (like SCSI). Persistent Reservations allow restricting +access to block devices to specific initiators in a shared storage +setup. + +This document gives a general overview of the support ioctl commands. +For a more detailed reference please refer the the SCSI Primary +Commands standard, specifically the section on Reservations and the +"PERSISTENT RESERVE IN" and "PERSISTENT RESERVE OUT" commands. + +All implementations are expected to ensure the reservations survive +a power loss and cover all connections in a multi path environment. +These behaviors are optional in SPC but will be automatically applied +by Linux. + + +The following types of reservations are supported: +-------------------------------------------------- + + - PR_WRITE_EXCLUSIVE + + Only the initiator that owns the reservation can write to the + device. Any initiator can read from the device. + + - PR_EXCLUSIVE_ACCESS + + Only the initiator that owns the reservation can access the + device. + + - PR_WRITE_EXCLUSIVE_REG_ONLY + + Only initiators with a registered key can write to the device, + Any initiator can read from the device. + + - PR_EXCLUSIVE_ACCESS_REG_ONLY + + Only initiators with a registered key can access the device. + + - PR_WRITE_EXCLUSIVE_ALL_REGS + + Only initiators with a registered key can write to the device, + Any initiator can read from the device. + All initiators with a registered key are considered reservation + holders. + Please reference the SPC spec on the meaning of a reservation + holder if you want to use this type. + + - PR_EXCLUSIVE_ACCESS_ALL_REGS + + Only initiators with a registered key can access the device. + All initiators with a registered key are considered reservation + holders. + Please reference the SPC spec on the meaning of a reservation + holder if you want to use this type. + + +The following ioctl are supported: +---------------------------------- + +1. IOC_PR_REGISTER + +This ioctl command registers a new reservation if the new_key argument +is non-null. If no existing reservation exists old_key must be zero, +if an existing reservation should be replaced old_key must contain +the old reservation key. + +If the new_key argument is 0 it unregisters the existing reservation passed +in old_key. + + +2. IOC_PR_RESERVE + +This ioctl command reserves the device and thus restricts access for other +devices based on the type argument. The key argument must be the existing +reservation key for the device as acquired by the IOC_PR_REGISTER, +IOC_PR_REGISTER_IGNORE, IOC_PR_PREEMPT or IOC_PR_PREEMPT_ABORT commands. + + +3. IOC_PR_RELEASE + +This ioctl command releases the reservation specified by key and flags +and thus removes any access restriction implied by it. + + +4. IOC_PR_PREEMPT + +This ioctl command releases the existing reservation referred to by +old_key and replaces it with a new reservation of type for the +reservation key new_key. + + +5. IOC_PR_PREEMPT_ABORT + +This ioctl command works like IOC_PR_PREEMPT except that it also aborts +any outstanding command sent over a connection identified by old_key. + +6. IOC_PR_CLEAR + +This ioctl command unregisters both key and any other reservation key +registered with the device and drops any existing reservation. + + +Flags +----- + +All the ioctls have a flag field. Currently only one flag is supported: + + - PR_FL_IGNORE_KEY + + Ignore the existing reservation key. This is commonly supported for + IOC_PR_REGISTER, and some implementation may support the flag for + IOC_PR_RESERVE. + +For all unknown flags the kernel will return -EOPNOTSUPP. diff --git a/Documentation/block/queue-sysfs.txt b/Documentation/block/queue-sysfs.txt new file mode 100644 index 000000000..2c1e67058 --- /dev/null +++ b/Documentation/block/queue-sysfs.txt @@ -0,0 +1,197 @@ +Queue sysfs files +================= + +This text file will detail the queue files that are located in the sysfs tree +for each block device. Note that stacked devices typically do not export +any settings, since their queue merely functions are a remapping target. +These files are the ones found in the /sys/block/xxx/queue/ directory. + +Files denoted with a RO postfix are readonly and the RW postfix means +read-write. + +add_random (RW) +---------------- +This file allows to turn off the disk entropy contribution. Default +value of this file is '1'(on). + +dax (RO) +-------- +This file indicates whether the device supports Direct Access (DAX), +used by CPU-addressable storage to bypass the pagecache. It shows '1' +if true, '0' if not. + +discard_granularity (RO) +----------------------- +This shows the size of internal allocation of the device in bytes, if +reported by the device. A value of '0' means device does not support +the discard functionality. + +discard_max_hw_bytes (RO) +---------------------- +Devices that support discard functionality may have internal limits on +the number of bytes that can be trimmed or unmapped in a single operation. +The discard_max_bytes parameter is set by the device driver to the maximum +number of bytes that can be discarded in a single operation. Discard +requests issued to the device must not exceed this limit. A discard_max_bytes +value of 0 means that the device does not support discard functionality. + +discard_max_bytes (RW) +---------------------- +While discard_max_hw_bytes is the hardware limit for the device, this +setting is the software limit. Some devices exhibit large latencies when +large discards are issued, setting this value lower will make Linux issue +smaller discards and potentially help reduce latencies induced by large +discard operations. + +hw_sector_size (RO) +------------------- +This is the hardware sector size of the device, in bytes. + +io_poll (RW) +------------ +When read, this file shows whether polling is enabled (1) or disabled +(0). Writing '0' to this file will disable polling for this device. +Writing any non-zero value will enable this feature. + +io_poll_delay (RW) +------------------ +If polling is enabled, this controls what kind of polling will be +performed. It defaults to -1, which is classic polling. In this mode, +the CPU will repeatedly ask for completions without giving up any time. +If set to 0, a hybrid polling mode is used, where the kernel will attempt +to make an educated guess at when the IO will complete. Based on this +guess, the kernel will put the process issuing IO to sleep for an amount +of time, before entering a classic poll loop. This mode might be a +little slower than pure classic polling, but it will be more efficient. +If set to a value larger than 0, the kernel will put the process issuing +IO to sleep for this amont of microseconds before entering classic +polling. + +iostats (RW) +------------- +This file is used to control (on/off) the iostats accounting of the +disk. + +logical_block_size (RO) +----------------------- +This is the logical block size of the device, in bytes. + +max_hw_sectors_kb (RO) +---------------------- +This is the maximum number of kilobytes supported in a single data transfer. + +max_integrity_segments (RO) +--------------------------- +When read, this file shows the max limit of integrity segments as +set by block layer which a hardware controller can handle. + +max_sectors_kb (RW) +------------------- +This is the maximum number of kilobytes that the block layer will allow +for a filesystem request. Must be smaller than or equal to the maximum +size allowed by the hardware. + +max_segments (RO) +----------------- +Maximum number of segments of the device. + +max_segment_size (RO) +--------------------- +Maximum segment size of the device. + +minimum_io_size (RO) +-------------------- +This is the smallest preferred IO size reported by the device. + +nomerges (RW) +------------- +This enables the user to disable the lookup logic involved with IO +merging requests in the block layer. By default (0) all merges are +enabled. When set to 1 only simple one-hit merges will be tried. When +set to 2 no merge algorithms will be tried (including one-hit or more +complex tree/hash lookups). + +nr_requests (RW) +---------------- +This controls how many requests may be allocated in the block layer for +read or write requests. Note that the total allocated number may be twice +this amount, since it applies only to reads or writes (not the accumulated +sum). + +To avoid priority inversion through request starvation, a request +queue maintains a separate request pool per each cgroup when +CONFIG_BLK_CGROUP is enabled, and this parameter applies to each such +per-block-cgroup request pool. IOW, if there are N block cgroups, +each request queue may have up to N request pools, each independently +regulated by nr_requests. + +optimal_io_size (RO) +-------------------- +This is the optimal IO size reported by the device. + +physical_block_size (RO) +------------------------ +This is the physical block size of device, in bytes. + +read_ahead_kb (RW) +------------------ +Maximum number of kilobytes to read-ahead for filesystems on this block +device. + +rotational (RW) +--------------- +This file is used to stat if the device is of rotational type or +non-rotational type. + +rq_affinity (RW) +---------------- +If this option is '1', the block layer will migrate request completions to the +cpu "group" that originally submitted the request. For some workloads this +provides a significant reduction in CPU cycles due to caching effects. + +For storage configurations that need to maximize distribution of completion +processing setting this option to '2' forces the completion to run on the +requesting cpu (bypassing the "group" aggregation logic). + +scheduler (RW) +-------------- +When read, this file will display the current and available IO schedulers +for this block device. The currently active IO scheduler will be enclosed +in [] brackets. Writing an IO scheduler name to this file will switch +control of this block device to that new IO scheduler. Note that writing +an IO scheduler name to this file will attempt to load that IO scheduler +module, if it isn't already present in the system. + +write_cache (RW) +---------------- +When read, this file will display whether the device has write back +caching enabled or not. It will return "write back" for the former +case, and "write through" for the latter. Writing to this file can +change the kernels view of the device, but it doesn't alter the +device state. This means that it might not be safe to toggle the +setting from "write back" to "write through", since that will also +eliminate cache flushes issued by the kernel. + +write_same_max_bytes (RO) +------------------------- +This is the number of bytes the device can write in a single write-same +command. A value of '0' means write-same is not supported by this +device. + +wb_lat_usec (RW) +---------------- +If the device is registered for writeback throttling, then this file shows +the target minimum read latency. If this latency is exceeded in a given +window of time (see wb_window_usec), then the writeback throttling will start +scaling back writes. Writing a value of '0' to this file disables the +feature. Writing a value of '-1' to this file resets the value to the +default setting. + +throttle_sample_time (RW) +------------------------- +This is the time window that blk-throttle samples data, in millisecond. +blk-throttle makes decision based on the samplings. Lower time means cgroups +have more smooth throughput, but higher CPU overhead. This exists only when +CONFIG_BLK_DEV_THROTTLING_LOW is enabled. + +Jens Axboe <jens.axboe@oracle.com>, February 2009 diff --git a/Documentation/block/request.txt b/Documentation/block/request.txt new file mode 100644 index 000000000..754e104ed --- /dev/null +++ b/Documentation/block/request.txt @@ -0,0 +1,88 @@ + +struct request documentation + +Jens Axboe <jens.axboe@oracle.com> 27/05/02 + +1.0 +Index + +2.0 Struct request members classification + + 2.1 struct request members explanation + +3.0 + + +2.0 +Short explanation of request members + +Classification flags: + + D driver member + B block layer member + I I/O scheduler member + +Unless an entry contains a D classification, a device driver must not access +this member. Some members may contain D classifications, but should only be +access through certain macros or functions (eg ->flags). + +<linux/blkdev.h> + +2.1 +Member Flag Comment +------ ---- ------- + +struct list_head queuelist BI Organization on various internal + queues + +void *elevator_private I I/O scheduler private data + +unsigned char cmd[16] D Driver can use this for setting up + a cdb before execution, see + blk_queue_prep_rq + +unsigned long flags DBI Contains info about data direction, + request type, etc. + +int rq_status D Request status bits + +kdev_t rq_dev DBI Target device + +int errors DB Error counts + +sector_t sector DBI Target location + +unsigned long hard_nr_sectors B Used to keep sector sane + +unsigned long nr_sectors DBI Total number of sectors in request + +unsigned long hard_nr_sectors B Used to keep nr_sectors sane + +unsigned short nr_phys_segments DB Number of physical scatter gather + segments in a request + +unsigned short nr_hw_segments DB Number of hardware scatter gather + segments in a request + +unsigned int current_nr_sectors DB Number of sectors in first segment + of request + +unsigned int hard_cur_sectors B Used to keep current_nr_sectors sane + +int tag DB TCQ tag, if assigned + +void *special D Free to be used by driver + +char *buffer D Map of first segment, also see + section on bouncing SECTION + +struct completion *waiting D Can be used by driver to get signalled + on request completion + +struct bio *bio DBI First bio in request + +struct bio *biotail DBI Last bio in request + +struct request_queue *q DB Request queue this request belongs to + +struct request_list *rl B Request list this request came from diff --git a/Documentation/block/stat.txt b/Documentation/block/stat.txt new file mode 100644 index 000000000..0aace9cc5 --- /dev/null +++ b/Documentation/block/stat.txt @@ -0,0 +1,86 @@ +Block layer statistics in /sys/block/<dev>/stat +=============================================== + +This file documents the contents of the /sys/block/<dev>/stat file. + +The stat file provides several statistics about the state of block +device <dev>. + +Q. Why are there multiple statistics in a single file? Doesn't sysfs + normally contain a single value per file? +A. By having a single file, the kernel can guarantee that the statistics + represent a consistent snapshot of the state of the device. If the + statistics were exported as multiple files containing one statistic + each, it would be impossible to guarantee that a set of readings + represent a single point in time. + +The stat file consists of a single line of text containing 11 decimal +values separated by whitespace. The fields are summarized in the +following table, and described in more detail below. + +Name units description +---- ----- ----------- +read I/Os requests number of read I/Os processed +read merges requests number of read I/Os merged with in-queue I/O +read sectors sectors number of sectors read +read ticks milliseconds total wait time for read requests +write I/Os requests number of write I/Os processed +write merges requests number of write I/Os merged with in-queue I/O +write sectors sectors number of sectors written +write ticks milliseconds total wait time for write requests +in_flight requests number of I/Os currently in flight +io_ticks milliseconds total time this block device has been active +time_in_queue milliseconds total wait time for all requests +discard I/Os requests number of discard I/Os processed +discard merges requests number of discard I/Os merged with in-queue I/O +discard sectors sectors number of sectors discarded +discard ticks milliseconds total wait time for discard requests + +read I/Os, write I/Os, discard I/0s +=================================== + +These values increment when an I/O request completes. + +read merges, write merges, discard merges +========================================= + +These values increment when an I/O request is merged with an +already-queued I/O request. + +read sectors, write sectors, discard_sectors +============================================ + +These values count the number of sectors read from, written to, or +discarded from this block device. The "sectors" in question are the +standard UNIX 512-byte sectors, not any device- or filesystem-specific +block size. The counters are incremented when the I/O completes. + +read ticks, write ticks, discard ticks +====================================== + +These values count the number of milliseconds that I/O requests have +waited on this block device. If there are multiple I/O requests waiting, +these values will increase at a rate greater than 1000/second; for +example, if 60 read requests wait for an average of 30 ms, the read_ticks +field will increase by 60*30 = 1800. + +in_flight +========= + +This value counts the number of I/O requests that have been issued to +the device driver but have not yet completed. It does not include I/O +requests that are in the queue but not yet issued to the device driver. + +io_ticks +======== + +This value counts the number of milliseconds during which the device has +had I/O requests queued. + +time_in_queue +============= + +This value counts the number of milliseconds that I/O requests have waited +on this block device. If there are multiple I/O requests waiting, this +value will increase as the product of the number of milliseconds times the +number of requests waiting (see "read ticks" above for an example). diff --git a/Documentation/block/switching-sched.txt b/Documentation/block/switching-sched.txt new file mode 100644 index 000000000..3b2612e34 --- /dev/null +++ b/Documentation/block/switching-sched.txt @@ -0,0 +1,37 @@ +To choose IO schedulers at boot time, use the argument 'elevator=deadline'. +'noop' and 'cfq' (the default) are also available. IO schedulers are assigned +globally at boot time only presently. + +Each io queue has a set of io scheduler tunables associated with it. These +tunables control how the io scheduler works. You can find these entries +in: + +/sys/block/<device>/queue/iosched + +assuming that you have sysfs mounted on /sys. If you don't have sysfs mounted, +you can do so by typing: + +# mount none /sys -t sysfs + +As of the Linux 2.6.10 kernel, it is now possible to change the +IO scheduler for a given block device on the fly (thus making it possible, +for instance, to set the CFQ scheduler for the system default, but +set a specific device to use the deadline or noop schedulers - which +can improve that device's throughput). + +To set a specific scheduler, simply do this: + +echo SCHEDNAME > /sys/block/DEV/queue/scheduler + +where SCHEDNAME is the name of a defined IO scheduler, and DEV is the +device name (hda, hdb, sga, or whatever you happen to have). + +The list of defined schedulers can be found by simply doing +a "cat /sys/block/DEV/queue/scheduler" - the list of valid names +will be displayed, with the currently selected scheduler in brackets: + +# cat /sys/block/hda/queue/scheduler +noop deadline [cfq] +# echo deadline > /sys/block/hda/queue/scheduler +# cat /sys/block/hda/queue/scheduler +noop [deadline] cfq diff --git a/Documentation/block/writeback_cache_control.txt b/Documentation/block/writeback_cache_control.txt new file mode 100644 index 000000000..8a6bdada5 --- /dev/null +++ b/Documentation/block/writeback_cache_control.txt @@ -0,0 +1,86 @@ + +Explicit volatile write back cache control +===================================== + +Introduction +------------ + +Many storage devices, especially in the consumer market, come with volatile +write back caches. That means the devices signal I/O completion to the +operating system before data actually has hit the non-volatile storage. This +behavior obviously speeds up various workloads, but it means the operating +system needs to force data out to the non-volatile storage when it performs +a data integrity operation like fsync, sync or an unmount. + +The Linux block layer provides two simple mechanisms that let filesystems +control the caching behavior of the storage device. These mechanisms are +a forced cache flush, and the Force Unit Access (FUA) flag for requests. + + +Explicit cache flushes +---------------------- + +The REQ_PREFLUSH flag can be OR ed into the r/w flags of a bio submitted from +the filesystem and will make sure the volatile cache of the storage device +has been flushed before the actual I/O operation is started. This explicitly +guarantees that previously completed write requests are on non-volatile +storage before the flagged bio starts. In addition the REQ_PREFLUSH flag can be +set on an otherwise empty bio structure, which causes only an explicit cache +flush without any dependent I/O. It is recommend to use +the blkdev_issue_flush() helper for a pure cache flush. + + +Forced Unit Access +----------------- + +The REQ_FUA flag can be OR ed into the r/w flags of a bio submitted from the +filesystem and will make sure that I/O completion for this request is only +signaled after the data has been committed to non-volatile storage. + + +Implementation details for filesystems +-------------------------------------- + +Filesystems can simply set the REQ_PREFLUSH and REQ_FUA bits and do not have to +worry if the underlying devices need any explicit cache flushing and how +the Forced Unit Access is implemented. The REQ_PREFLUSH and REQ_FUA flags +may both be set on a single bio. + + +Implementation details for make_request_fn based block drivers +-------------------------------------------------------------- + +These drivers will always see the REQ_PREFLUSH and REQ_FUA bits as they sit +directly below the submit_bio interface. For remapping drivers the REQ_FUA +bits need to be propagated to underlying devices, and a global flush needs +to be implemented for bios with the REQ_PREFLUSH bit set. For real device +drivers that do not have a volatile cache the REQ_PREFLUSH and REQ_FUA bits +on non-empty bios can simply be ignored, and REQ_PREFLUSH requests without +data can be completed successfully without doing any work. Drivers for +devices with volatile caches need to implement the support for these +flags themselves without any help from the block layer. + + +Implementation details for request_fn based block drivers +-------------------------------------------------------------- + +For devices that do not support volatile write caches there is no driver +support required, the block layer completes empty REQ_PREFLUSH requests before +entering the driver and strips off the REQ_PREFLUSH and REQ_FUA bits from +requests that have a payload. For devices with volatile write caches the +driver needs to tell the block layer that it supports flushing caches by +doing: + + blk_queue_write_cache(sdkp->disk->queue, true, false); + +and handle empty REQ_OP_FLUSH requests in its prep_fn/request_fn. Note that +REQ_PREFLUSH requests with a payload are automatically turned into a sequence +of an empty REQ_OP_FLUSH request followed by the actual write by the block +layer. For devices that also support the FUA bit the block layer needs +to be told to pass through the REQ_FUA bit using: + + blk_queue_write_cache(sdkp->disk->queue, true, true); + +and the driver must handle write requests that have the REQ_FUA bit set +in prep_fn/request_fn. If the FUA bit is not natively supported the block +layer turns it into an empty REQ_OP_FLUSH request after the actual write. diff --git a/Documentation/blockdev/00-INDEX b/Documentation/blockdev/00-INDEX new file mode 100644 index 000000000..c08df56dd --- /dev/null +++ b/Documentation/blockdev/00-INDEX @@ -0,0 +1,18 @@ +00-INDEX + - this file +README.DAC960 + - info on Mylex DAC960/DAC1100 PCI RAID Controller Driver for Linux. +cciss.txt + - info, major/minor #'s for Compaq's SMART Array Controllers. +cpqarray.txt + - info on using Compaq's SMART2 Intelligent Disk Array Controllers. +floppy.txt + - notes and driver options for the floppy disk driver. +mflash.txt + - info on mGine m(g)flash driver for linux. +nbd.txt + - info on a TCP implementation of a network block device. +paride.txt + - information about the parallel port IDE subsystem. +ramdisk.txt + - short guide on how to set up and use the RAM disk. diff --git a/Documentation/blockdev/README.DAC960 b/Documentation/blockdev/README.DAC960 new file mode 100644 index 000000000..bd85fb9dc --- /dev/null +++ b/Documentation/blockdev/README.DAC960 @@ -0,0 +1,756 @@ + Linux Driver for Mylex DAC960/AcceleRAID/eXtremeRAID PCI RAID Controllers + + Version 2.2.11 for Linux 2.2.19 + Version 2.4.11 for Linux 2.4.12 + + PRODUCTION RELEASE + + 11 October 2001 + + Leonard N. Zubkoff + Dandelion Digital + lnz@dandelion.com + + Copyright 1998-2001 by Leonard N. Zubkoff <lnz@dandelion.com> + + + INTRODUCTION + +Mylex, Inc. designs and manufactures a variety of high performance PCI RAID +controllers. Mylex Corporation is located at 34551 Ardenwood Blvd., Fremont, +California 94555, USA and can be reached at 510.796.6100 or on the World Wide +Web at http://www.mylex.com. Mylex Technical Support can be reached by +electronic mail at mylexsup@us.ibm.com, by voice at 510.608.2400, or by FAX at +510.745.7715. Contact information for offices in Europe and Japan is available +on their Web site. + +The latest information on Linux support for DAC960 PCI RAID Controllers, as +well as the most recent release of this driver, will always be available from +my Linux Home Page at URL "http://www.dandelion.com/Linux/". The Linux DAC960 +driver supports all current Mylex PCI RAID controllers including the new +eXtremeRAID 2000/3000 and AcceleRAID 352/170/160 models which have an entirely +new firmware interface from the older eXtremeRAID 1100, AcceleRAID 150/200/250, +and DAC960PJ/PG/PU/PD/PL. See below for a complete controller list as well as +minimum firmware version requirements. For simplicity, in most places this +documentation refers to DAC960 generically rather than explicitly listing all +the supported models. + +Driver bug reports should be sent via electronic mail to "lnz@dandelion.com". +Please include with the bug report the complete configuration messages reported +by the driver at startup, along with any subsequent system messages relevant to +the controller's operation, and a detailed description of your system's +hardware configuration. Driver bugs are actually quite rare; if you encounter +problems with disks being marked offline, for example, please contact Mylex +Technical Support as the problem is related to the hardware configuration +rather than the Linux driver. + +Please consult the RAID controller documentation for detailed information +regarding installation and configuration of the controllers. This document +primarily provides information specific to the Linux support. + + + DRIVER FEATURES + +The DAC960 RAID controllers are supported solely as high performance RAID +controllers, not as interfaces to arbitrary SCSI devices. The Linux DAC960 +driver operates at the block device level, the same level as the SCSI and IDE +drivers. Unlike other RAID controllers currently supported on Linux, the +DAC960 driver is not dependent on the SCSI subsystem, and hence avoids all the +complexity and unnecessary code that would be associated with an implementation +as a SCSI driver. The DAC960 driver is designed for as high a performance as +possible with no compromises or extra code for compatibility with lower +performance devices. The DAC960 driver includes extensive error logging and +online configuration management capabilities. Except for initial configuration +of the controller and adding new disk drives, most everything can be handled +from Linux while the system is operational. + +The DAC960 driver is architected to support up to 8 controllers per system. +Each DAC960 parallel SCSI controller can support up to 15 disk drives per +channel, for a maximum of 60 drives on a four channel controller; the fibre +channel eXtremeRAID 3000 controller supports up to 125 disk drives per loop for +a total of 250 drives. The drives installed on a controller are divided into +one or more "Drive Groups", and then each Drive Group is subdivided further +into 1 to 32 "Logical Drives". Each Logical Drive has a specific RAID Level +and caching policy associated with it, and it appears to Linux as a single +block device. Logical Drives are further subdivided into up to 7 partitions +through the normal Linux and PC disk partitioning schemes. Logical Drives are +also known as "System Drives", and Drive Groups are also called "Packs". Both +terms are in use in the Mylex documentation; I have chosen to standardize on +the more generic "Logical Drive" and "Drive Group". + +DAC960 RAID disk devices are named in the style of the obsolete Device File +System (DEVFS). The device corresponding to Logical Drive D on Controller C +is referred to as /dev/rd/cCdD, and the partitions are called /dev/rd/cCdDp1 +through /dev/rd/cCdDp7. For example, partition 3 of Logical Drive 5 on +Controller 2 is referred to as /dev/rd/c2d5p3. Note that unlike with SCSI +disks the device names will not change in the event of a disk drive failure. +The DAC960 driver is assigned major numbers 48 - 55 with one major number per +controller. The 8 bits of minor number are divided into 5 bits for the Logical +Drive and 3 bits for the partition. + + + SUPPORTED DAC960/AcceleRAID/eXtremeRAID PCI RAID CONTROLLERS + +The following list comprises the supported DAC960, AcceleRAID, and eXtremeRAID +PCI RAID Controllers as of the date of this document. It is recommended that +anyone purchasing a Mylex PCI RAID Controller not in the following table +contact the author beforehand to verify that it is or will be supported. + +eXtremeRAID 3000 + 1 Wide Ultra-2/LVD SCSI channel + 2 External Fibre FC-AL channels + 233MHz StrongARM SA 110 Processor + 64 Bit 33MHz PCI (backward compatible with 32 Bit PCI slots) + 32MB/64MB ECC SDRAM Memory + +eXtremeRAID 2000 + 4 Wide Ultra-160 LVD SCSI channels + 233MHz StrongARM SA 110 Processor + 64 Bit 33MHz PCI (backward compatible with 32 Bit PCI slots) + 32MB/64MB ECC SDRAM Memory + +AcceleRAID 352 + 2 Wide Ultra-160 LVD SCSI channels + 100MHz Intel i960RN RISC Processor + 64 Bit 33MHz PCI (backward compatible with 32 Bit PCI slots) + 32MB/64MB ECC SDRAM Memory + +AcceleRAID 170 + 1 Wide Ultra-160 LVD SCSI channel + 100MHz Intel i960RM RISC Processor + 16MB/32MB/64MB ECC SDRAM Memory + +AcceleRAID 160 (AcceleRAID 170LP) + 1 Wide Ultra-160 LVD SCSI channel + 100MHz Intel i960RS RISC Processor + Built in 16M ECC SDRAM Memory + PCI Low Profile Form Factor - fit for 2U height + +eXtremeRAID 1100 (DAC1164P) + 3 Wide Ultra-2/LVD SCSI channels + 233MHz StrongARM SA 110 Processor + 64 Bit 33MHz PCI (backward compatible with 32 Bit PCI slots) + 16MB/32MB/64MB Parity SDRAM Memory with Battery Backup + +AcceleRAID 250 (DAC960PTL1) + Uses onboard Symbios SCSI chips on certain motherboards + Also includes one onboard Wide Ultra-2/LVD SCSI Channel + 66MHz Intel i960RD RISC Processor + 4MB/8MB/16MB/32MB/64MB/128MB ECC EDO Memory + +AcceleRAID 200 (DAC960PTL0) + Uses onboard Symbios SCSI chips on certain motherboards + Includes no onboard SCSI Channels + 66MHz Intel i960RD RISC Processor + 4MB/8MB/16MB/32MB/64MB/128MB ECC EDO Memory + +AcceleRAID 150 (DAC960PRL) + Uses onboard Symbios SCSI chips on certain motherboards + Also includes one onboard Wide Ultra-2/LVD SCSI Channel + 33MHz Intel i960RP RISC Processor + 4MB Parity EDO Memory + +DAC960PJ 1/2/3 Wide Ultra SCSI-3 Channels + 66MHz Intel i960RD RISC Processor + 4MB/8MB/16MB/32MB/64MB/128MB ECC EDO Memory + +DAC960PG 1/2/3 Wide Ultra SCSI-3 Channels + 33MHz Intel i960RP RISC Processor + 4MB/8MB ECC EDO Memory + +DAC960PU 1/2/3 Wide Ultra SCSI-3 Channels + Intel i960CF RISC Processor + 4MB/8MB EDRAM or 2MB/4MB/8MB/16MB/32MB DRAM Memory + +DAC960PD 1/2/3 Wide Fast SCSI-2 Channels + Intel i960CF RISC Processor + 4MB/8MB EDRAM or 2MB/4MB/8MB/16MB/32MB DRAM Memory + +DAC960PL 1/2/3 Wide Fast SCSI-2 Channels + Intel i960 RISC Processor + 2MB/4MB/8MB/16MB/32MB DRAM Memory + +DAC960P 1/2/3 Wide Fast SCSI-2 Channels + Intel i960 RISC Processor + 2MB/4MB/8MB/16MB/32MB DRAM Memory + +For the eXtremeRAID 2000/3000 and AcceleRAID 352/170/160, firmware version +6.00-01 or above is required. + +For the eXtremeRAID 1100, firmware version 5.06-0-52 or above is required. + +For the AcceleRAID 250, 200, and 150, firmware version 4.06-0-57 or above is +required. + +For the DAC960PJ and DAC960PG, firmware version 4.06-0-00 or above is required. + +For the DAC960PU, DAC960PD, DAC960PL, and DAC960P, either firmware version +3.51-0-04 or above is required (for dual Flash ROM controllers), or firmware +version 2.73-0-00 or above is required (for single Flash ROM controllers) + +Please note that not all SCSI disk drives are suitable for use with DAC960 +controllers, and only particular firmware versions of any given model may +actually function correctly. Similarly, not all motherboards have a BIOS that +properly initializes the AcceleRAID 250, AcceleRAID 200, AcceleRAID 150, +DAC960PJ, and DAC960PG because the Intel i960RD/RP is a multi-function device. +If in doubt, contact Mylex RAID Technical Support (mylexsup@us.ibm.com) to +verify compatibility. Mylex makes available a hard disk compatibility list at +http://www.mylex.com/support/hdcomp/hd-lists.html. + + + DRIVER INSTALLATION + +This distribution was prepared for Linux kernel version 2.2.19 or 2.4.12. + +To install the DAC960 RAID driver, you may use the following commands, +replacing "/usr/src" with wherever you keep your Linux kernel source tree: + + cd /usr/src + tar -xvzf DAC960-2.2.11.tar.gz (or DAC960-2.4.11.tar.gz) + mv README.DAC960 linux/Documentation + mv DAC960.[ch] linux/drivers/block + patch -p0 < DAC960.patch (if DAC960.patch is included) + cd linux + make config + make bzImage (or zImage) + +Then install "arch/x86/boot/bzImage" or "arch/x86/boot/zImage" as your +standard kernel, run lilo if appropriate, and reboot. + +To create the necessary devices in /dev, the "make_rd" script included in +"DAC960-Utilities.tar.gz" from http://www.dandelion.com/Linux/ may be used. +LILO 21 and FDISK v2.9 include DAC960 support; also included in this archive +are patches to LILO 20 and FDISK v2.8 that add DAC960 support, along with +statically linked executables of LILO and FDISK. This modified version of LILO +will allow booting from a DAC960 controller and/or mounting the root file +system from a DAC960. + +Red Hat Linux 6.0 and SuSE Linux 6.1 include support for Mylex PCI RAID +controllers. Installing directly onto a DAC960 may be problematic from other +Linux distributions until their installation utilities are updated. + + + INSTALLATION NOTES + +Before installing Linux or adding DAC960 logical drives to an existing Linux +system, the controller must first be configured to provide one or more logical +drives using the BIOS Configuration Utility or DACCF. Please note that since +there are only at most 6 usable partitions on each logical drive, systems +requiring more partitions should subdivide a drive group into multiple logical +drives, each of which can have up to 6 usable partitions. Also, note that with +large disk arrays it is advisable to enable the 8GB BIOS Geometry (255/63) +rather than accepting the default 2GB BIOS Geometry (128/32); failing to so do +will cause the logical drive geometry to have more than 65535 cylinders which +will make it impossible for FDISK to be used properly. The 8GB BIOS Geometry +can be enabled by configuring the DAC960 BIOS, which is accessible via Alt-M +during the BIOS initialization sequence. + +For maximum performance and the most efficient E2FSCK performance, it is +recommended that EXT2 file systems be built with a 4KB block size and 16 block +stride to match the DAC960 controller's 64KB default stripe size. The command +"mke2fs -b 4096 -R stride=16 <device>" is appropriate. Unless there will be a +large number of small files on the file systems, it is also beneficial to add +the "-i 16384" option to increase the bytes per inode parameter thereby +reducing the file system metadata. Finally, on systems that will only be run +with Linux 2.2 or later kernels it is beneficial to enable sparse superblocks +with the "-s 1" option. + + + DAC960 ANNOUNCEMENTS MAILING LIST + +The DAC960 Announcements Mailing List provides a forum for informing Linux +users of new driver releases and other announcements regarding Linux support +for DAC960 PCI RAID Controllers. To join the mailing list, send a message to +"dac960-announce-request@dandelion.com" with the line "subscribe" in the +message body. + + + CONTROLLER CONFIGURATION AND STATUS MONITORING + +The DAC960 RAID controllers running firmware 4.06 or above include a Background +Initialization facility so that system downtime is minimized both for initial +installation and subsequent configuration of additional storage. The BIOS +Configuration Utility (accessible via Alt-R during the BIOS initialization +sequence) is used to quickly configure the controller, and then the logical +drives that have been created are available for immediate use even while they +are still being initialized by the controller. The primary need for online +configuration and status monitoring is then to avoid system downtime when disk +drives fail and must be replaced. Mylex's online monitoring and configuration +utilities are being ported to Linux and will become available at some point in +the future. Note that with a SAF-TE (SCSI Accessed Fault-Tolerant Enclosure) +enclosure, the controller is able to rebuild failed drives automatically as +soon as a drive replacement is made available. + +The primary interfaces for controller configuration and status monitoring are +special files created in the /proc/rd/... hierarchy along with the normal +system console logging mechanism. Whenever the system is operating, the DAC960 +driver queries each controller for status information every 10 seconds, and +checks for additional conditions every 60 seconds. The initial status of each +controller is always available for controller N in /proc/rd/cN/initial_status, +and the current status as of the last status monitoring query is available in +/proc/rd/cN/current_status. In addition, status changes are also logged by the +driver to the system console and will appear in the log files maintained by +syslog. The progress of asynchronous rebuild or consistency check operations +is also available in /proc/rd/cN/current_status, and progress messages are +logged to the system console at most every 60 seconds. + +Starting with the 2.2.3/2.0.3 versions of the driver, the status information +available in /proc/rd/cN/initial_status and /proc/rd/cN/current_status has been +augmented to include the vendor, model, revision, and serial number (if +available) for each physical device found connected to the controller: + +***** DAC960 RAID Driver Version 2.2.3 of 19 August 1999 ***** +Copyright 1998-1999 by Leonard N. Zubkoff <lnz@dandelion.com> +Configuring Mylex DAC960PRL PCI RAID Controller + Firmware Version: 4.07-0-07, Channels: 1, Memory Size: 16MB + PCI Bus: 1, Device: 4, Function: 1, I/O Address: Unassigned + PCI Address: 0xFE300000 mapped at 0xA0800000, IRQ Channel: 21 + Controller Queue Depth: 128, Maximum Blocks per Command: 128 + Driver Queue Depth: 127, Maximum Scatter/Gather Segments: 33 + Stripe Size: 64KB, Segment Size: 8KB, BIOS Geometry: 255/63 + SAF-TE Enclosure Management Enabled + Physical Devices: + 0:0 Vendor: IBM Model: DRVS09D Revision: 0270 + Serial Number: 68016775HA + Disk Status: Online, 17928192 blocks + 0:1 Vendor: IBM Model: DRVS09D Revision: 0270 + Serial Number: 68004E53HA + Disk Status: Online, 17928192 blocks + 0:2 Vendor: IBM Model: DRVS09D Revision: 0270 + Serial Number: 13013935HA + Disk Status: Online, 17928192 blocks + 0:3 Vendor: IBM Model: DRVS09D Revision: 0270 + Serial Number: 13016897HA + Disk Status: Online, 17928192 blocks + 0:4 Vendor: IBM Model: DRVS09D Revision: 0270 + Serial Number: 68019905HA + Disk Status: Online, 17928192 blocks + 0:5 Vendor: IBM Model: DRVS09D Revision: 0270 + Serial Number: 68012753HA + Disk Status: Online, 17928192 blocks + 0:6 Vendor: ESG-SHV Model: SCA HSBP M6 Revision: 0.61 + Logical Drives: + /dev/rd/c0d0: RAID-5, Online, 89640960 blocks, Write Thru + No Rebuild or Consistency Check in Progress + +To simplify the monitoring process for custom software, the special file +/proc/rd/status returns "OK" when all DAC960 controllers in the system are +operating normally and no failures have occurred, or "ALERT" if any logical +drives are offline or critical or any non-standby physical drives are dead. + +Configuration commands for controller N are available via the special file +/proc/rd/cN/user_command. A human readable command can be written to this +special file to initiate a configuration operation, and the results of the +operation can then be read back from the special file in addition to being +logged to the system console. The shell command sequence + + echo "<configuration-command>" > /proc/rd/c0/user_command + cat /proc/rd/c0/user_command + +is typically used to execute configuration commands. The configuration +commands are: + + flush-cache + + The "flush-cache" command flushes the controller's cache. The system + automatically flushes the cache at shutdown or if the driver module is + unloaded, so this command is only needed to be certain a write back cache + is flushed to disk before the system is powered off by a command to a UPS. + Note that the flush-cache command also stops an asynchronous rebuild or + consistency check, so it should not be used except when the system is being + halted. + + kill <channel>:<target-id> + + The "kill" command marks the physical drive <channel>:<target-id> as DEAD. + This command is provided primarily for testing, and should not be used + during normal system operation. + + make-online <channel>:<target-id> + + The "make-online" command changes the physical drive <channel>:<target-id> + from status DEAD to status ONLINE. In cases where multiple physical drives + have been killed simultaneously, this command may be used to bring all but + one of them back online, after which a rebuild to the final drive is + necessary. + + Warning: make-online should only be used on a dead physical drive that is + an active part of a drive group, never on a standby drive. The command + should never be used on a dead drive that is part of a critical logical + drive; rebuild should be used if only a single drive is dead. + + make-standby <channel>:<target-id> + + The "make-standby" command changes physical drive <channel>:<target-id> + from status DEAD to status STANDBY. It should only be used in cases where + a dead drive was replaced after an automatic rebuild was performed onto a + standby drive. It cannot be used to add a standby drive to the controller + configuration if one was not created initially; the BIOS Configuration + Utility must be used for that currently. + + rebuild <channel>:<target-id> + + The "rebuild" command initiates an asynchronous rebuild onto physical drive + <channel>:<target-id>. It should only be used when a dead drive has been + replaced. + + check-consistency <logical-drive-number> + + The "check-consistency" command initiates an asynchronous consistency check + of <logical-drive-number> with automatic restoration. It can be used + whenever it is desired to verify the consistency of the redundancy + information. + + cancel-rebuild + cancel-consistency-check + + The "cancel-rebuild" and "cancel-consistency-check" commands cancel any + rebuild or consistency check operations previously initiated. + + + EXAMPLE I - DRIVE FAILURE WITHOUT A STANDBY DRIVE + +The following annotated logs demonstrate the controller configuration and and +online status monitoring capabilities of the Linux DAC960 Driver. The test +configuration comprises 6 1GB Quantum Atlas I disk drives on two channels of a +DAC960PJ controller. The physical drives are configured into a single drive +group without a standby drive, and the drive group has been configured into two +logical drives, one RAID-5 and one RAID-6. Note that these logs are from an +earlier version of the driver and the messages have changed somewhat with newer +releases, but the functionality remains similar. First, here is the current +status of the RAID configuration: + +gwynedd:/u/lnz# cat /proc/rd/c0/current_status +***** DAC960 RAID Driver Version 2.0.0 of 23 March 1999 ***** +Copyright 1998-1999 by Leonard N. Zubkoff <lnz@dandelion.com> +Configuring Mylex DAC960PJ PCI RAID Controller + Firmware Version: 4.06-0-08, Channels: 3, Memory Size: 8MB + PCI Bus: 0, Device: 19, Function: 1, I/O Address: Unassigned + PCI Address: 0xFD4FC000 mapped at 0x8807000, IRQ Channel: 9 + Controller Queue Depth: 128, Maximum Blocks per Command: 128 + Driver Queue Depth: 127, Maximum Scatter/Gather Segments: 33 + Stripe Size: 64KB, Segment Size: 8KB, BIOS Geometry: 255/63 + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Online, 2201600 blocks + 1:2 - Disk: Online, 2201600 blocks + 1:3 - Disk: Online, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Online, 5498880 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Online, 3305472 blocks, Write Thru + No Rebuild or Consistency Check in Progress + +gwynedd:/u/lnz# cat /proc/rd/status +OK + +The above messages indicate that everything is healthy, and /proc/rd/status +returns "OK" indicating that there are no problems with any DAC960 controller +in the system. For demonstration purposes, while I/O is active Physical Drive +1:1 is now disconnected, simulating a drive failure. The failure is noted by +the driver within 10 seconds of the controller's having detected it, and the +driver logs the following console status messages indicating that Logical +Drives 0 and 1 are now CRITICAL as a result of Physical Drive 1:1 being DEAD: + +DAC960#0: Physical Drive 1:2 Error Log: Sense Key = 6, ASC = 29, ASCQ = 02 +DAC960#0: Physical Drive 1:3 Error Log: Sense Key = 6, ASC = 29, ASCQ = 02 +DAC960#0: Physical Drive 1:1 killed because of timeout on SCSI command +DAC960#0: Physical Drive 1:1 is now DEAD +DAC960#0: Logical Drive 0 (/dev/rd/c0d0) is now CRITICAL +DAC960#0: Logical Drive 1 (/dev/rd/c0d1) is now CRITICAL + +The Sense Keys logged here are just Check Condition / Unit Attention conditions +arising from a SCSI bus reset that is forced by the controller during its error +recovery procedures. Concurrently with the above, the driver status available +from /proc/rd also reflects the drive failure. The status message in +/proc/rd/status has changed from "OK" to "ALERT": + +gwynedd:/u/lnz# cat /proc/rd/status +ALERT + +and /proc/rd/c0/current_status has been updated: + +gwynedd:/u/lnz# cat /proc/rd/c0/current_status + ... + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Dead, 2201600 blocks + 1:2 - Disk: Online, 2201600 blocks + 1:3 - Disk: Online, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Critical, 5498880 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Critical, 3305472 blocks, Write Thru + No Rebuild or Consistency Check in Progress + +Since there are no standby drives configured, the system can continue to access +the logical drives in a performance degraded mode until the failed drive is +replaced and a rebuild operation completed to restore the redundancy of the +logical drives. Once Physical Drive 1:1 is replaced with a properly +functioning drive, or if the physical drive was killed without having failed +(e.g., due to electrical problems on the SCSI bus), the user can instruct the +controller to initiate a rebuild operation onto the newly replaced drive: + +gwynedd:/u/lnz# echo "rebuild 1:1" > /proc/rd/c0/user_command +gwynedd:/u/lnz# cat /proc/rd/c0/user_command +Rebuild of Physical Drive 1:1 Initiated + +The echo command instructs the controller to initiate an asynchronous rebuild +operation onto Physical Drive 1:1, and the status message that results from the +operation is then available for reading from /proc/rd/c0/user_command, as well +as being logged to the console by the driver. + +Within 10 seconds of this command the driver logs the initiation of the +asynchronous rebuild operation: + +DAC960#0: Rebuild of Physical Drive 1:1 Initiated +DAC960#0: Physical Drive 1:1 Error Log: Sense Key = 6, ASC = 29, ASCQ = 01 +DAC960#0: Physical Drive 1:1 is now WRITE-ONLY +DAC960#0: Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 1% completed + +and /proc/rd/c0/current_status is updated: + +gwynedd:/u/lnz# cat /proc/rd/c0/current_status + ... + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Write-Only, 2201600 blocks + 1:2 - Disk: Online, 2201600 blocks + 1:3 - Disk: Online, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Critical, 5498880 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Critical, 3305472 blocks, Write Thru + Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 6% completed + +As the rebuild progresses, the current status in /proc/rd/c0/current_status is +updated every 10 seconds: + +gwynedd:/u/lnz# cat /proc/rd/c0/current_status + ... + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Write-Only, 2201600 blocks + 1:2 - Disk: Online, 2201600 blocks + 1:3 - Disk: Online, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Critical, 5498880 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Critical, 3305472 blocks, Write Thru + Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 15% completed + +and every minute a progress message is logged to the console by the driver: + +DAC960#0: Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 32% completed +DAC960#0: Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 63% completed +DAC960#0: Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 94% completed +DAC960#0: Rebuild in Progress: Logical Drive 1 (/dev/rd/c0d1) 94% completed + +Finally, the rebuild completes successfully. The driver logs the status of the +logical and physical drives and the rebuild completion: + +DAC960#0: Rebuild Completed Successfully +DAC960#0: Physical Drive 1:1 is now ONLINE +DAC960#0: Logical Drive 0 (/dev/rd/c0d0) is now ONLINE +DAC960#0: Logical Drive 1 (/dev/rd/c0d1) is now ONLINE + +/proc/rd/c0/current_status is updated: + +gwynedd:/u/lnz# cat /proc/rd/c0/current_status + ... + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Online, 2201600 blocks + 1:2 - Disk: Online, 2201600 blocks + 1:3 - Disk: Online, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Online, 5498880 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Online, 3305472 blocks, Write Thru + Rebuild Completed Successfully + +and /proc/rd/status indicates that everything is healthy once again: + +gwynedd:/u/lnz# cat /proc/rd/status +OK + + + EXAMPLE II - DRIVE FAILURE WITH A STANDBY DRIVE + +The following annotated logs demonstrate the controller configuration and and +online status monitoring capabilities of the Linux DAC960 Driver. The test +configuration comprises 6 1GB Quantum Atlas I disk drives on two channels of a +DAC960PJ controller. The physical drives are configured into a single drive +group with a standby drive, and the drive group has been configured into two +logical drives, one RAID-5 and one RAID-6. Note that these logs are from an +earlier version of the driver and the messages have changed somewhat with newer +releases, but the functionality remains similar. First, here is the current +status of the RAID configuration: + +gwynedd:/u/lnz# cat /proc/rd/c0/current_status +***** DAC960 RAID Driver Version 2.0.0 of 23 March 1999 ***** +Copyright 1998-1999 by Leonard N. Zubkoff <lnz@dandelion.com> +Configuring Mylex DAC960PJ PCI RAID Controller + Firmware Version: 4.06-0-08, Channels: 3, Memory Size: 8MB + PCI Bus: 0, Device: 19, Function: 1, I/O Address: Unassigned + PCI Address: 0xFD4FC000 mapped at 0x8807000, IRQ Channel: 9 + Controller Queue Depth: 128, Maximum Blocks per Command: 128 + Driver Queue Depth: 127, Maximum Scatter/Gather Segments: 33 + Stripe Size: 64KB, Segment Size: 8KB, BIOS Geometry: 255/63 + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Online, 2201600 blocks + 1:2 - Disk: Online, 2201600 blocks + 1:3 - Disk: Standby, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Online, 4399104 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Online, 2754560 blocks, Write Thru + No Rebuild or Consistency Check in Progress + +gwynedd:/u/lnz# cat /proc/rd/status +OK + +The above messages indicate that everything is healthy, and /proc/rd/status +returns "OK" indicating that there are no problems with any DAC960 controller +in the system. For demonstration purposes, while I/O is active Physical Drive +1:2 is now disconnected, simulating a drive failure. The failure is noted by +the driver within 10 seconds of the controller's having detected it, and the +driver logs the following console status messages: + +DAC960#0: Physical Drive 1:1 Error Log: Sense Key = 6, ASC = 29, ASCQ = 02 +DAC960#0: Physical Drive 1:3 Error Log: Sense Key = 6, ASC = 29, ASCQ = 02 +DAC960#0: Physical Drive 1:2 killed because of timeout on SCSI command +DAC960#0: Physical Drive 1:2 is now DEAD +DAC960#0: Physical Drive 1:2 killed because it was removed +DAC960#0: Logical Drive 0 (/dev/rd/c0d0) is now CRITICAL +DAC960#0: Logical Drive 1 (/dev/rd/c0d1) is now CRITICAL + +Since a standby drive is configured, the controller automatically begins +rebuilding onto the standby drive: + +DAC960#0: Physical Drive 1:3 is now WRITE-ONLY +DAC960#0: Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 4% completed + +Concurrently with the above, the driver status available from /proc/rd also +reflects the drive failure and automatic rebuild. The status message in +/proc/rd/status has changed from "OK" to "ALERT": + +gwynedd:/u/lnz# cat /proc/rd/status +ALERT + +and /proc/rd/c0/current_status has been updated: + +gwynedd:/u/lnz# cat /proc/rd/c0/current_status + ... + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Online, 2201600 blocks + 1:2 - Disk: Dead, 2201600 blocks + 1:3 - Disk: Write-Only, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Critical, 4399104 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Critical, 2754560 blocks, Write Thru + Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 4% completed + +As the rebuild progresses, the current status in /proc/rd/c0/current_status is +updated every 10 seconds: + +gwynedd:/u/lnz# cat /proc/rd/c0/current_status + ... + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Online, 2201600 blocks + 1:2 - Disk: Dead, 2201600 blocks + 1:3 - Disk: Write-Only, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Critical, 4399104 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Critical, 2754560 blocks, Write Thru + Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 40% completed + +and every minute a progress message is logged on the console by the driver: + +DAC960#0: Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 40% completed +DAC960#0: Rebuild in Progress: Logical Drive 0 (/dev/rd/c0d0) 76% completed +DAC960#0: Rebuild in Progress: Logical Drive 1 (/dev/rd/c0d1) 66% completed +DAC960#0: Rebuild in Progress: Logical Drive 1 (/dev/rd/c0d1) 84% completed + +Finally, the rebuild completes successfully. The driver logs the status of the +logical and physical drives and the rebuild completion: + +DAC960#0: Rebuild Completed Successfully +DAC960#0: Physical Drive 1:3 is now ONLINE +DAC960#0: Logical Drive 0 (/dev/rd/c0d0) is now ONLINE +DAC960#0: Logical Drive 1 (/dev/rd/c0d1) is now ONLINE + +/proc/rd/c0/current_status is updated: + +***** DAC960 RAID Driver Version 2.0.0 of 23 March 1999 ***** +Copyright 1998-1999 by Leonard N. Zubkoff <lnz@dandelion.com> +Configuring Mylex DAC960PJ PCI RAID Controller + Firmware Version: 4.06-0-08, Channels: 3, Memory Size: 8MB + PCI Bus: 0, Device: 19, Function: 1, I/O Address: Unassigned + PCI Address: 0xFD4FC000 mapped at 0x8807000, IRQ Channel: 9 + Controller Queue Depth: 128, Maximum Blocks per Command: 128 + Driver Queue Depth: 127, Maximum Scatter/Gather Segments: 33 + Stripe Size: 64KB, Segment Size: 8KB, BIOS Geometry: 255/63 + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Online, 2201600 blocks + 1:2 - Disk: Dead, 2201600 blocks + 1:3 - Disk: Online, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Online, 4399104 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Online, 2754560 blocks, Write Thru + Rebuild Completed Successfully + +and /proc/rd/status indicates that everything is healthy once again: + +gwynedd:/u/lnz# cat /proc/rd/status +OK + +Note that the absence of a viable standby drive does not create an "ALERT" +status. Once dead Physical Drive 1:2 has been replaced, the controller must be +told that this has occurred and that the newly replaced drive should become the +new standby drive: + +gwynedd:/u/lnz# echo "make-standby 1:2" > /proc/rd/c0/user_command +gwynedd:/u/lnz# cat /proc/rd/c0/user_command +Make Standby of Physical Drive 1:2 Succeeded + +The echo command instructs the controller to make Physical Drive 1:2 into a +standby drive, and the status message that results from the operation is then +available for reading from /proc/rd/c0/user_command, as well as being logged to +the console by the driver. Within 60 seconds of this command the driver logs: + +DAC960#0: Physical Drive 1:2 Error Log: Sense Key = 6, ASC = 29, ASCQ = 01 +DAC960#0: Physical Drive 1:2 is now STANDBY +DAC960#0: Make Standby of Physical Drive 1:2 Succeeded + +and /proc/rd/c0/current_status is updated: + +gwynedd:/u/lnz# cat /proc/rd/c0/current_status + ... + Physical Devices: + 0:1 - Disk: Online, 2201600 blocks + 0:2 - Disk: Online, 2201600 blocks + 0:3 - Disk: Online, 2201600 blocks + 1:1 - Disk: Online, 2201600 blocks + 1:2 - Disk: Standby, 2201600 blocks + 1:3 - Disk: Online, 2201600 blocks + Logical Drives: + /dev/rd/c0d0: RAID-5, Online, 4399104 blocks, Write Thru + /dev/rd/c0d1: RAID-6, Online, 2754560 blocks, Write Thru + Rebuild Completed Successfully diff --git a/Documentation/blockdev/drbd/DRBD-8.3-data-packets.svg b/Documentation/blockdev/drbd/DRBD-8.3-data-packets.svg new file mode 100644 index 000000000..f87cfa0dc --- /dev/null +++ b/Documentation/blockdev/drbd/DRBD-8.3-data-packets.svg @@ -0,0 +1,588 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> +<svg + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + version="1.0" + width="210mm" + height="297mm" + viewBox="0 0 21000 29700" + id="svg2" + style="fill-rule:evenodd"> + <defs + id="defs4" /> + <g + id="Default" + style="visibility:visible"> + <desc + id="desc180">Master slide</desc> + </g> + <path + d="M 11999,8601 L 11899,8301 L 12099,8301 L 11999,8601 z" + id="path193" + style="fill:#008000;visibility:visible" /> + <path + d="M 11999,7801 L 11999,8361" + id="path197" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 7999,10401 L 7899,10101 L 8099,10101 L 7999,10401 z" + id="path209" + style="fill:#008000;visibility:visible" /> + <path + d="M 7999,9601 L 7999,10161" + id="path213" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 11999,7801 L 11685,7840 L 11724,7644 L 11999,7801 z" + id="path225" + style="fill:#008000;visibility:visible" /> + <path + d="M 7999,7001 L 11764,7754" + id="path229" + style="fill:none;stroke:#008000;visibility:visible" /> + <g + transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,-1244.4792,1416.5139)" + id="g245" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <text + id="text247"> + <tspan + x="9139 9368 9579 9808 9986 10075 10252 10481 10659 10837 10909" + y="9284" + id="tspan249">RSDataReply</tspan> + </text> + </g> + <path + d="M 7999,9601 L 8281,9458 L 8311,9655 L 7999,9601 z" + id="path259" + style="fill:#008000;visibility:visible" /> + <path + d="M 11999,9001 L 8236,9565" + id="path263" + style="fill:none;stroke:#008000;visibility:visible" /> + <g + transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,1620.9382,-1639.4947)" + id="g279" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <text + id="text281"> + <tspan + x="8743 8972 9132 9310 9573 9801 10013 10242 10419 10597 10775 10953 11114" + y="7023" + id="tspan283">CsumRSRequest</tspan> + </text> + </g> + <text + id="text297" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4034 4263 4440 4703 4881 5042 5219 5397 5503 5681 5842 6003 6180 6341 6519 6625 6803 6980 7158 7336 7497 7586 7692" + y="5707" + id="tspan299">w_make_resync_request()</tspan> + </text> + <text + id="text313" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12199 12305 12483 12644 12821 12893 13054 13232 13410 13638 13816 13905 14083 14311 14489 14667 14845 15023 15184 15272 15378" + y="7806" + id="tspan315">receive_DataRequest()</tspan> + </text> + <text + id="text329" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12199 12377 12483 12660 12838 13016 13194 13372 13549 13621 13799 13977 14083 14261 14438 14616 14794 14955 15133 15294 15399" + y="8606" + id="tspan331">drbd_endio_read_sec()</tspan> + </text> + <text + id="text345" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12191 12420 12597 12775 12953 13131 13309 13486 13664 13825 13986 14164 14426 14604 14710 14871 15049 15154 15332 15510 15616" + y="9007" + id="tspan347">w_e_end_csum_rs_req()</tspan> + </text> + <text + id="text361" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4444 4550 4728 4889 5066 5138 5299 5477 5655 5883 6095 6324 6501 6590 6768 6997 7175 7352 7424 7585 7691" + y="9507" + id="tspan363">receive_RSDataReply()</tspan> + </text> + <text + id="text377" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4457 4635 4741 4918 5096 5274 5452 5630 5807 5879 6057 6235 6464 6569 6641 6730 6908 7086 7247 7425 7585 7691" + y="10407" + id="tspan379">drbd_endio_write_sec()</tspan> + </text> + <text + id="text393" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4647 4825 5003 5180 5358 5536 5714 5820 5997 6158 6319 6497 6658 6836 7013 7085 7263 7424 7585 7691" + y="10907" + id="tspan395">e_end_resync_block()</tspan> + </text> + <path + d="M 11999,11601 L 11685,11640 L 11724,11444 L 11999,11601 z" + id="path405" + style="fill:#000080;visibility:visible" /> + <path + d="M 7999,10801 L 11764,11554" + id="path409" + style="fill:none;stroke:#000080;visibility:visible" /> + <g + transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,2434.7562,-1674.649)" + id="g425" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <text + id="text427"> + <tspan + x="9320 9621 9726 9798 9887 10065 10277 10438" + y="10943" + id="tspan429">WriteAck</tspan> + </text> + </g> + <text + id="text443" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12199 12377 12555 12644 12821 13033 13105 13283 13444 13604 13816 13977 14138 14244" + y="11559" + id="tspan445">got_BlockAck()</tspan> + </text> + <text + id="text459" + style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="7999 8304 8541 8778 8990 9201 9413 9650 10001 10120 10357 10594 10806 11043 11280 11398 11703 11940 12152 12364 12601 12812 12931 13049 13261 13498 13710 13947 14065 14302 14540 14658 14777 14870 15107 15225 15437 15649 15886" + y="4877" + id="tspan461">Checksum based Resync, case not in sync</tspan> + </text> + <text + id="text475" + style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="6961 7266 7571 7854 8159 8299 8536 8654 8891 9010 9247 9484 9603 9840 9958 10077 10170 10407" + y="2806" + id="tspan477">DRBD-8.3 data flow</tspan> + </text> + <text + id="text491" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="5190 5419 5596 5774 5952 6113 6291 6468 6646 6824 6985 7146 7324 7586 7692" + y="7005" + id="tspan493">w_e_send_csum()</tspan> + </text> + <path + d="M 11999,17601 L 11899,17301 L 12099,17301 L 11999,17601 z" + id="path503" + style="fill:#008000;visibility:visible" /> + <path + d="M 11999,16801 L 11999,17361" + id="path507" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 11999,16801 L 11685,16840 L 11724,16644 L 11999,16801 z" + id="path519" + style="fill:#008000;visibility:visible" /> + <path + d="M 7999,16001 L 11764,16754" + id="path523" + style="fill:none;stroke:#008000;visibility:visible" /> + <g + transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,-2539.5806,1529.3491)" + id="g539" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <text + id="text541"> + <tspan + x="9269 9498 9709 9798 9959 10048 10226 10437 10598 10776" + y="18265" + id="tspan543">RSIsInSync</tspan> + </text> + </g> + <path + d="M 7999,18601 L 8281,18458 L 8311,18655 L 7999,18601 z" + id="path553" + style="fill:#000080;visibility:visible" /> + <path + d="M 11999,18001 L 8236,18565" + id="path557" + style="fill:none;stroke:#000080;visibility:visible" /> + <g + transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,3461.4027,-1449.3012)" + id="g573" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <text + id="text575"> + <tspan + x="8743 8972 9132 9310 9573 9801 10013 10242 10419 10597 10775 10953 11114" + y="16023" + id="tspan577">CsumRSRequest</tspan> + </text> + </g> + <text + id="text591" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12199 12305 12483 12644 12821 12893 13054 13232 13410 13638 13816 13905 14083 14311 14489 14667 14845 15023 15184 15272 15378" + y="16806" + id="tspan593">receive_DataRequest()</tspan> + </text> + <text + id="text607" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12199 12377 12483 12660 12838 13016 13194 13372 13549 13621 13799 13977 14083 14261 14438 14616 14794 14955 15133 15294 15399" + y="17606" + id="tspan609">drbd_endio_read_sec()</tspan> + </text> + <text + id="text623" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12191 12420 12597 12775 12953 13131 13309 13486 13664 13825 13986 14164 14426 14604 14710 14871 15049 15154 15332 15510 15616" + y="18007" + id="tspan625">w_e_end_csum_rs_req()</tspan> + </text> + <text + id="text639" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="5735 5913 6091 6180 6357 6446 6607 6696 6874 7085 7246 7424 7585 7691" + y="18507" + id="tspan641">got_IsInSync()</tspan> + </text> + <text + id="text655" + style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="7999 8304 8541 8778 8990 9201 9413 9650 10001 10120 10357 10594 10806 11043 11280 11398 11703 11940 12152 12364 12601 12812 12931 13049 13261 13498 13710 13947 14065 14159 14396 14514 14726 14937 15175" + y="13877" + id="tspan657">Checksum based Resync, case in sync</tspan> + </text> + <path + d="M 12000,24601 L 11900,24301 L 12100,24301 L 12000,24601 z" + id="path667" + style="fill:#008000;visibility:visible" /> + <path + d="M 12000,23801 L 12000,24361" + id="path671" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 8000,26401 L 7900,26101 L 8100,26101 L 8000,26401 z" + id="path683" + style="fill:#008000;visibility:visible" /> + <path + d="M 8000,25601 L 8000,26161" + id="path687" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 12000,23801 L 11686,23840 L 11725,23644 L 12000,23801 z" + id="path699" + style="fill:#008000;visibility:visible" /> + <path + d="M 8000,23001 L 11765,23754" + id="path703" + style="fill:none;stroke:#008000;visibility:visible" /> + <g + transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,-3543.8452,1630.5143)" + id="g719" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <text + id="text721"> + <tspan + x="9464 9710 9921 10150 10328 10505 10577" + y="25236" + id="tspan723">OVReply</tspan> + </text> + </g> + <path + d="M 8000,25601 L 8282,25458 L 8312,25655 L 8000,25601 z" + id="path733" + style="fill:#008000;visibility:visible" /> + <path + d="M 12000,25001 L 8237,25565" + id="path737" + style="fill:none;stroke:#008000;visibility:visible" /> + <g + transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,4918.2801,-1381.2128)" + id="g753" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <text + id="text755"> + <tspan + x="9142 9388 9599 9828 10006 10183 10361 10539 10700" + y="23106" + id="tspan757">OVRequest</tspan> + </text> + </g> + <text + id="text771" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12200 12306 12484 12645 12822 12894 13055 13233 13411 13656 13868 14097 14274 14452 14630 14808 14969 15058 15163" + y="23806" + id="tspan773">receive_OVRequest()</tspan> + </text> + <text + id="text787" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12200 12378 12484 12661 12839 13017 13195 13373 13550 13622 13800 13978 14084 14262 14439 14617 14795 14956 15134 15295 15400" + y="24606" + id="tspan789">drbd_endio_read_sec()</tspan> + </text> + <text + id="text803" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12192 12421 12598 12776 12954 13132 13310 13487 13665 13843 14004 14182 14288 14465 14643 14749" + y="25007" + id="tspan805">w_e_end_ov_req()</tspan> + </text> + <text + id="text819" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="5101 5207 5385 5546 5723 5795 5956 6134 6312 6557 6769 6998 7175 7353 7425 7586 7692" + y="25507" + id="tspan821">receive_OVReply()</tspan> + </text> + <text + id="text835" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4492 4670 4776 4953 5131 5309 5487 5665 5842 5914 6092 6270 6376 6554 6731 6909 7087 7248 7426 7587 7692" + y="26407" + id="tspan837">drbd_endio_read_sec()</tspan> + </text> + <text + id="text851" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4902 5131 5308 5486 5664 5842 6020 6197 6375 6553 6714 6892 6998 7175 7353 7425 7586 7692" + y="26907" + id="tspan853">w_e_end_ov_reply()</tspan> + </text> + <path + d="M 12000,27601 L 11686,27640 L 11725,27444 L 12000,27601 z" + id="path863" + style="fill:#000080;visibility:visible" /> + <path + d="M 8000,26801 L 11765,27554" + id="path867" + style="fill:none;stroke:#000080;visibility:visible" /> + <g + transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,5704.1907,-1328.312)" + id="g883" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <text + id="text885"> + <tspan + x="9279 9525 9736 9965 10143 10303 10481 10553" + y="26935" + id="tspan887">OVResult</tspan> + </text> + </g> + <text + id="text901" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12200 12378 12556 12645 12822 13068 13280 13508 13686 13847 14025 14097 14185 14291" + y="27559" + id="tspan903">got_OVResult()</tspan> + </text> + <text + id="text917" + style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="8000 8330 8567 8660 8754 8991 9228 9346 9558 9795 9935 10028 10146" + y="21877" + id="tspan919">Online verify</tspan> + </text> + <text + id="text933" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4641 4870 5047 5310 5488 5649 5826 6004 6182 6343 6521 6626 6804 6982 7160 7338 7499 7587 7693" + y="23005" + id="tspan935">w_make_ov_request()</tspan> + </text> + <path + d="M 8000,6500 L 7900,6200 L 8100,6200 L 8000,6500 z" + id="path945" + style="fill:#008000;visibility:visible" /> + <path + d="M 8000,5700 L 8000,6260" + id="path949" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 3900,5500 L 3700,5500 L 3700,11000 L 3900,11000" + id="path961" + style="fill:none;stroke:#000000;visibility:visible" /> + <path + d="M 3900,14500 L 3700,14500 L 3700,18600 L 3900,18600" + id="path973" + style="fill:none;stroke:#000000;visibility:visible" /> + <path + d="M 3900,22800 L 3700,22800 L 3700,26900 L 3900,26900" + id="path985" + style="fill:none;stroke:#000000;visibility:visible" /> + <text + id="text1001" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4492 4670 4776 4953 5131 5309 5487 5665 5842 5914 6092 6270 6376 6554 6731 6909 7087 7248 7426 7587 7692" + y="6506" + id="tspan1003">drbd_endio_read_sec()</tspan> + </text> + <text + id="text1017" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4034 4263 4440 4703 4881 5042 5219 5397 5503 5681 5842 6003 6180 6341 6519 6625 6803 6980 7158 7336 7497 7586 7692" + y="14708" + id="tspan1019">w_make_resync_request()</tspan> + </text> + <text + id="text1033" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="5190 5419 5596 5774 5952 6113 6291 6468 6646 6824 6985 7146 7324 7586 7692" + y="16006" + id="tspan1035">w_e_send_csum()</tspan> + </text> + <path + d="M 8000,15501 L 7900,15201 L 8100,15201 L 8000,15501 z" + id="path1045" + style="fill:#008000;visibility:visible" /> + <path + d="M 8000,14701 L 8000,15261" + id="path1049" + style="fill:none;stroke:#008000;visibility:visible" /> + <text + id="text1065" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4492 4670 4776 4953 5131 5309 5487 5665 5842 5914 6092 6270 6376 6554 6731 6909 7087 7248 7426 7587 7692" + y="15507" + id="tspan1067">drbd_endio_read_sec()</tspan> + </text> + <path + d="M 16100,9000 L 16300,9000 L 16300,7500 L 16100,7500" + id="path1077" + style="fill:none;stroke:#000000;visibility:visible" /> + <path + d="M 16100,18000 L 16300,18000 L 16300,16500 L 16100,16500" + id="path1089" + style="fill:none;stroke:#000000;visibility:visible" /> + <path + d="M 16100,25000 L 16300,25000 L 16300,23500 L 16100,23500" + id="path1101" + style="fill:none;stroke:#000000;visibility:visible" /> + <text + id="text1117" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="2026 2132 2293 2471 2648 2826 3004 3076 3254 3431 3503 3681 3787" + y="5402" + id="tspan1119">rs_begin_io()</tspan> + </text> + <text + id="text1133" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="2027 2133 2294 2472 2649 2827 3005 3077 3255 3432 3504 3682 3788" + y="14402" + id="tspan1135">rs_begin_io()</tspan> + </text> + <text + id="text1149" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="2026 2132 2293 2471 2648 2826 3004 3076 3254 3431 3503 3681 3787" + y="22602" + id="tspan1151">rs_begin_io()</tspan> + </text> + <text + id="text1165" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="1426 1532 1693 1871 2031 2209 2472 2649 2721 2899 2988 3166 3344 3416 3593 3699" + y="11302" + id="tspan1167">rs_complete_io()</tspan> + </text> + <text + id="text1181" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="1526 1632 1793 1971 2131 2309 2572 2749 2821 2999 3088 3266 3444 3516 3693 3799" + y="18931" + id="tspan1183">rs_complete_io()</tspan> + </text> + <text + id="text1197" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="1526 1632 1793 1971 2131 2309 2572 2749 2821 2999 3088 3266 3444 3516 3693 3799" + y="27231" + id="tspan1199">rs_complete_io()</tspan> + </text> + <text + id="text1213" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="16126 16232 16393 16571 16748 16926 17104 17176 17354 17531 17603 17781 17887" + y="7402" + id="tspan1215">rs_begin_io()</tspan> + </text> + <text + id="text1229" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="16127 16233 16394 16572 16749 16927 17105 17177 17355 17532 17604 17782 17888" + y="16331" + id="tspan1231">rs_begin_io()</tspan> + </text> + <text + id="text1245" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="16127 16233 16394 16572 16749 16927 17105 17177 17355 17532 17604 17782 17888" + y="23302" + id="tspan1247">rs_begin_io()</tspan> + </text> + <text + id="text1261" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="16115 16221 16382 16560 16720 16898 17161 17338 17410 17588 17677 17855 18033 18105 18282 18388" + y="9302" + id="tspan1263">rs_complete_io()</tspan> + </text> + <text + id="text1277" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="16115 16221 16382 16560 16720 16898 17161 17338 17410 17588 17677 17855 18033 18105 18282 18388" + y="18331" + id="tspan1279">rs_complete_io()</tspan> + </text> + <text + id="text1293" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="16126 16232 16393 16571 16731 16909 17172 17349 17421 17599 17688 17866 18044 18116 18293 18399" + y="25302" + id="tspan1295">rs_complete_io()</tspan> + </text> +</svg> diff --git a/Documentation/blockdev/drbd/DRBD-data-packets.svg b/Documentation/blockdev/drbd/DRBD-data-packets.svg new file mode 100644 index 000000000..48a1e2165 --- /dev/null +++ b/Documentation/blockdev/drbd/DRBD-data-packets.svg @@ -0,0 +1,459 @@ +<?xml version="1.0" encoding="UTF-8" standalone="no"?> +<!-- Created with Inkscape (http://www.inkscape.org/) --> +<svg + xmlns:svg="http://www.w3.org/2000/svg" + xmlns="http://www.w3.org/2000/svg" + version="1.0" + width="210mm" + height="297mm" + viewBox="0 0 21000 29700" + id="svg2" + style="fill-rule:evenodd"> + <defs + id="defs4" /> + <g + id="Default" + style="visibility:visible"> + <desc + id="desc176">Master slide</desc> + </g> + <path + d="M 11999,19601 L 11899,19301 L 12099,19301 L 11999,19601 z" + id="path189" + style="fill:#008000;visibility:visible" /> + <path + d="M 11999,18801 L 11999,19361" + id="path193" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 7999,21401 L 7899,21101 L 8099,21101 L 7999,21401 z" + id="path205" + style="fill:#008000;visibility:visible" /> + <path + d="M 7999,20601 L 7999,21161" + id="path209" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 11999,18801 L 11685,18840 L 11724,18644 L 11999,18801 z" + id="path221" + style="fill:#008000;visibility:visible" /> + <path + d="M 7999,18001 L 11764,18754" + id="path225" + style="fill:none;stroke:#008000;visibility:visible" /> + <text + x="-3023.845" + y="1106.8124" + transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,0,0)" + id="text243" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="6115.1553 6344.1553 6555.1553 6784.1553 6962.1553 7051.1553 7228.1553 7457.1553 7635.1553 7813.1553 7885.1553" + y="21390.812" + id="tspan245">RSDataReply</tspan> + </text> + <path + d="M 7999,20601 L 8281,20458 L 8311,20655 L 7999,20601 z" + id="path255" + style="fill:#008000;visibility:visible" /> + <path + d="M 11999,20001 L 8236,20565" + id="path259" + style="fill:none;stroke:#008000;visibility:visible" /> + <text + x="3502.5356" + y="-2184.6621" + transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,0,0)" + id="text277" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12321.536 12550.536 12761.536 12990.536 13168.536 13257.536 13434.536 13663.536 13841.536 14019.536 14196.536 14374.536 14535.536" + y="15854.338" + id="tspan279">RSDataRequest</tspan> + </text> + <text + id="text293" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4034 4263 4440 4703 4881 5042 5219 5397 5503 5681 5842 6003 6180 6341 6519 6625 6803 6980 7158 7336 7497 7586 7692" + y="17807" + id="tspan295">w_make_resync_request()</tspan> + </text> + <text + id="text309" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12199 12305 12483 12644 12821 12893 13054 13232 13410 13638 13816 13905 14083 14311 14489 14667 14845 15023 15184 15272 15378" + y="18806" + id="tspan311">receive_DataRequest()</tspan> + </text> + <text + id="text325" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12199 12377 12483 12660 12838 13016 13194 13372 13549 13621 13799 13977 14083 14261 14438 14616 14794 14955 15133 15294 15399" + y="19606" + id="tspan327">drbd_endio_read_sec()</tspan> + </text> + <text + id="text341" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12191 12420 12597 12775 12953 13131 13309 13486 13664 13770 13931 14109 14287 14375 14553 14731 14837 15015 15192 15298" + y="20007" + id="tspan343">w_e_end_rsdata_req()</tspan> + </text> + <text + id="text357" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4444 4550 4728 4889 5066 5138 5299 5477 5655 5883 6095 6324 6501 6590 6768 6997 7175 7352 7424 7585 7691" + y="20507" + id="tspan359">receive_RSDataReply()</tspan> + </text> + <text + id="text373" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4457 4635 4741 4918 5096 5274 5452 5630 5807 5879 6057 6235 6464 6569 6641 6730 6908 7086 7247 7425 7585 7691" + y="21407" + id="tspan375">drbd_endio_write_sec()</tspan> + </text> + <text + id="text389" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4647 4825 5003 5180 5358 5536 5714 5820 5997 6158 6319 6497 6658 6836 7013 7085 7263 7424 7585 7691" + y="21907" + id="tspan391">e_end_resync_block()</tspan> + </text> + <path + d="M 11999,22601 L 11685,22640 L 11724,22444 L 11999,22601 z" + id="path401" + style="fill:#000080;visibility:visible" /> + <path + d="M 7999,21801 L 11764,22554" + id="path405" + style="fill:none;stroke:#000080;visibility:visible" /> + <text + x="4290.3008" + y="-2369.6162" + transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,0,0)" + id="text423" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="13610.301 13911.301 14016.301 14088.301 14177.301 14355.301 14567.301 14728.301" + y="19573.385" + id="tspan425">WriteAck</tspan> + </text> + <text + id="text439" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12199 12377 12555 12644 12821 13033 13105 13283 13444 13604 13816 13977 14138 14244" + y="22559" + id="tspan441">got_BlockAck()</tspan> + </text> + <text + id="text455" + style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="7999 8304 8541 8753 8964 9201 9413 9531 9769 9862 10099 10310 10522 10734 10852 10971 11208 11348 11585 11822" + y="16877" + id="tspan457">Resync blocks, 4-32K</tspan> + </text> + <path + d="M 12000,7601 L 11900,7301 L 12100,7301 L 12000,7601 z" + id="path467" + style="fill:#008000;visibility:visible" /> + <path + d="M 12000,6801 L 12000,7361" + id="path471" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 12000,6801 L 11686,6840 L 11725,6644 L 12000,6801 z" + id="path483" + style="fill:#008000;visibility:visible" /> + <path + d="M 8000,6001 L 11765,6754" + id="path487" + style="fill:none;stroke:#008000;visibility:visible" /> + <text + x="-1288.1796" + y="1279.7666" + transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,0,0)" + id="text505" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="8174.8208 8475.8203 8580.8203 8652.8203 8741.8203 8919.8203 9131.8203 9292.8203" + y="9516.7666" + id="tspan507">WriteAck</tspan> + </text> + <path + d="M 8000,8601 L 8282,8458 L 8312,8655 L 8000,8601 z" + id="path517" + style="fill:#000080;visibility:visible" /> + <path + d="M 12000,8001 L 8237,8565" + id="path521" + style="fill:none;stroke:#000080;visibility:visible" /> + <text + x="1065.6655" + y="-2097.7664" + transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,0,0)" + id="text539" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="10682.666 10911.666 11088.666 11177.666" + y="4107.2339" + id="tspan541">Data</tspan> + </text> + <text + id="text555" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4746 4924 5030 5207 5385 5563 5826 6003 6164 6342 6520 6626 6803 6981 7159 7337 7498 7587 7692" + y="5505" + id="tspan557">drbd_make_request()</tspan> + </text> + <text + id="text571" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12200 12306 12484 12645 12822 12894 13055 13233 13411 13639 13817 13906 14084 14190" + y="6806" + id="tspan573">receive_Data()</tspan> + </text> + <text + id="text587" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12200 12378 12484 12661 12839 13017 13195 13373 13550 13622 13800 13978 14207 14312 14384 14473 14651 14829 14990 15168 15328 15434" + y="7606" + id="tspan589">drbd_endio_write_sec()</tspan> + </text> + <text + id="text603" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12192 12370 12548 12725 12903 13081 13259 13437 13509 13686 13847 14008 14114" + y="8007" + id="tspan605">e_end_block()</tspan> + </text> + <text + id="text619" + style="font-size:318px;font-weight:400;fill:#000080;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="5647 5825 6003 6092 6269 6481 6553 6731 6892 7052 7264 7425 7586 7692" + y="8606" + id="tspan621">got_BlockAck()</tspan> + </text> + <text + id="text635" + style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="8000 8305 8542 8779 9016 9109 9346 9486 9604 9956 10049 10189 10328 10565 10705 10942 11179 11298 11603 11742 11835 11954 12191 12310 12428 12665 12902 13139 13279 13516 13753" + y="4877" + id="tspan637">Regular mirrored write, 512-32K</tspan> + </text> + <text + id="text651" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="5381 5610 5787 5948 6126 6304 6482 6659 6837 7015 7087 7265 7426 7587 7692" + y="6003" + id="tspan653">w_send_dblock()</tspan> + </text> + <path + d="M 8000,6800 L 7900,6500 L 8100,6500 L 8000,6800 z" + id="path663" + style="fill:#008000;visibility:visible" /> + <path + d="M 8000,6000 L 8000,6560" + id="path667" + style="fill:none;stroke:#008000;visibility:visible" /> + <text + id="text683" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4602 4780 4886 5063 5241 5419 5597 5775 5952 6024 6202 6380 6609 6714 6786 6875 7053 7231 7409 7515 7587 7692" + y="6905" + id="tspan685">drbd_endio_write_pri()</tspan> + </text> + <path + d="M 12000,13602 L 11900,13302 L 12100,13302 L 12000,13602 z" + id="path695" + style="fill:#008000;visibility:visible" /> + <path + d="M 12000,12802 L 12000,13362" + id="path699" + style="fill:none;stroke:#008000;visibility:visible" /> + <path + d="M 12000,12802 L 11686,12841 L 11725,12645 L 12000,12802 z" + id="path711" + style="fill:#008000;visibility:visible" /> + <path + d="M 8000,12002 L 11765,12755" + id="path715" + style="fill:none;stroke:#008000;visibility:visible" /> + <text + x="-2155.5266" + y="1201.5964" + transform="matrix(0.9895258,-0.1443562,0.1443562,0.9895258,0,0)" + id="text733" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="7202.4736 7431.4736 7608.4736 7697.4736 7875.4736 8104.4736 8282.4736 8459.4736 8531.4736" + y="15454.597" + id="tspan735">DataReply</tspan> + </text> + <path + d="M 8000,14602 L 8282,14459 L 8312,14656 L 8000,14602 z" + id="path745" + style="fill:#008000;visibility:visible" /> + <path + d="M 12000,14002 L 8237,14566" + id="path749" + style="fill:none;stroke:#008000;visibility:visible" /> + <text + x="2280.3804" + y="-2103.2141" + transform="matrix(0.9788674,0.2044961,-0.2044961,0.9788674,0,0)" + id="text767" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="11316.381 11545.381 11722.381 11811.381 11989.381 12218.381 12396.381 12573.381 12751.381 12929.381 13090.381" + y="9981.7861" + id="tspan769">DataRequest</tspan> + </text> + <text + id="text783" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="4746 4924 5030 5207 5385 5563 5826 6003 6164 6342 6520 6626 6803 6981 7159 7337 7498 7587 7692" + y="11506" + id="tspan785">drbd_make_request()</tspan> + </text> + <text + id="text799" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12200 12306 12484 12645 12822 12894 13055 13233 13411 13639 13817 13906 14084 14312 14490 14668 14846 15024 15185 15273 15379" + y="12807" + id="tspan801">receive_DataRequest()</tspan> + </text> + <text + id="text815" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12200 12378 12484 12661 12839 13017 13195 13373 13550 13622 13800 13978 14084 14262 14439 14617 14795 14956 15134 15295 15400" + y="13607" + id="tspan817">drbd_endio_read_sec()</tspan> + </text> + <text + id="text831" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="12192 12421 12598 12776 12954 13132 13310 13487 13665 13843 14021 14110 14288 14465 14571 14749 14927 15033" + y="14008" + id="tspan833">w_e_end_data_req()</tspan> + </text> + <g + id="g835" + style="visibility:visible"> + <desc + id="desc837">Drawing</desc> + <text + id="text847" + style="font-size:318px;font-weight:400;fill:#008000;font-family:Helvetica embedded"> + <tspan + x="4885 4991 5169 5330 5507 5579 5740 5918 6096 6324 6502 6591 6769 6997 7175 7353 7425 7586 7692" + y="14607" + id="tspan849">receive_DataReply()</tspan> + </text> + </g> + <text + id="text863" + style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="8000 8305 8398 8610 8821 8914 9151 9363 9575 9693 9833 10070 10307 10544 10663 10781 11018 11255 11493 11632 11869 12106" + y="10878" + id="tspan865">Diskless read, 512-32K</tspan> + </text> + <text + id="text879" + style="font-size:318px;font-weight:400;fill:#008000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="5029 5258 5435 5596 5774 5952 6130 6307 6413 6591 6769 6947 7125 7230 7408 7586 7692" + y="12004" + id="tspan881">w_send_read_req()</tspan> + </text> + <text + id="text895" + style="font-size:423px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="6961 7266 7571 7854 8159 8278 8515 8633 8870 9107 9226 9463 9581 9700 9793 10030" + y="2806" + id="tspan897">DRBD 8 data flow</tspan> + </text> + <path + d="M 3900,5300 L 3700,5300 L 3700,7000 L 3900,7000" + id="path907" + style="fill:none;stroke:#000000;visibility:visible" /> + <path + d="M 3900,17600 L 3700,17600 L 3700,22000 L 3900,22000" + id="path919" + style="fill:none;stroke:#000000;visibility:visible" /> + <path + d="M 16100,20000 L 16300,20000 L 16300,18500 L 16100,18500" + id="path931" + style="fill:none;stroke:#000000;visibility:visible" /> + <text + id="text947" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="2126 2304 2376 2554 2731 2909 3087 3159 3337 3515 3587 3764 3870" + y="5202" + id="tspan949">al_begin_io()</tspan> + </text> + <text + id="text963" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="1632 1810 1882 2060 2220 2398 2661 2839 2910 3088 3177 3355 3533 3605 3783 3888" + y="7331" + id="tspan965">al_complete_io()</tspan> + </text> + <text + id="text979" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="2126 2232 2393 2571 2748 2926 3104 3176 3354 3531 3603 3781 3887" + y="17431" + id="tspan981">rs_begin_io()</tspan> + </text> + <text + id="text995" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="1626 1732 1893 2071 2231 2409 2672 2849 2921 3099 3188 3366 3544 3616 3793 3899" + y="22331" + id="tspan997">rs_complete_io()</tspan> + </text> + <text + id="text1011" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="16027 16133 16294 16472 16649 16827 17005 17077 17255 17432 17504 17682 17788" + y="18402" + id="tspan1013">rs_begin_io()</tspan> + </text> + <text + id="text1027" + style="font-size:318px;font-weight:400;fill:#000000;visibility:visible;font-family:Helvetica embedded"> + <tspan + x="16115 16221 16382 16560 16720 16898 17161 17338 17410 17588 17677 17855 18033 18105 18282 18388" + y="20331" + id="tspan1029">rs_complete_io()</tspan> + </text> +</svg> diff --git a/Documentation/blockdev/drbd/README.txt b/Documentation/blockdev/drbd/README.txt new file mode 100644 index 000000000..627b0a1bf --- /dev/null +++ b/Documentation/blockdev/drbd/README.txt @@ -0,0 +1,16 @@ +Description + + DRBD is a shared-nothing, synchronously replicated block device. It + is designed to serve as a building block for high availability + clusters and in this context, is a "drop-in" replacement for shared + storage. Simplistically, you could see it as a network RAID 1. + + Please visit http://www.drbd.org to find out more. + +The here included files are intended to help understand the implementation + +DRBD-8.3-data-packets.svg, DRBD-data-packets.svg + relates some functions, and write packets. + +conn-states-8.dot, disk-states-8.dot, node-states-8.dot + The sub graphs of DRBD's state transitions diff --git a/Documentation/blockdev/drbd/conn-states-8.dot b/Documentation/blockdev/drbd/conn-states-8.dot new file mode 100644 index 000000000..025e8cf5e --- /dev/null +++ b/Documentation/blockdev/drbd/conn-states-8.dot @@ -0,0 +1,18 @@ +digraph conn_states { + StandAllone -> WFConnection [ label = "ioctl_set_net()" ] + WFConnection -> Unconnected [ label = "unable to bind()" ] + WFConnection -> WFReportParams [ label = "in connect() after accept" ] + WFReportParams -> StandAllone [ label = "checks in receive_param()" ] + WFReportParams -> Connected [ label = "in receive_param()" ] + WFReportParams -> WFBitMapS [ label = "sync_handshake()" ] + WFReportParams -> WFBitMapT [ label = "sync_handshake()" ] + WFBitMapS -> SyncSource [ label = "receive_bitmap()" ] + WFBitMapT -> SyncTarget [ label = "receive_bitmap()" ] + SyncSource -> Connected + SyncTarget -> Connected + SyncSource -> PausedSyncS + SyncTarget -> PausedSyncT + PausedSyncS -> SyncSource + PausedSyncT -> SyncTarget + Connected -> WFConnection [ label = "* on network error" ] +} diff --git a/Documentation/blockdev/drbd/data-structure-v9.txt b/Documentation/blockdev/drbd/data-structure-v9.txt new file mode 100644 index 000000000..1e52a0e32 --- /dev/null +++ b/Documentation/blockdev/drbd/data-structure-v9.txt @@ -0,0 +1,38 @@ +This describes the in kernel data structure for DRBD-9. Starting with +Linux v3.14 we are reorganizing DRBD to use this data structure. + +Basic Data Structure +==================== + +A node has a number of DRBD resources. Each such resource has a number of +devices (aka volumes) and connections to other nodes ("peer nodes"). Each DRBD +device is represented by a block device locally. + +The DRBD objects are interconnected to form a matrix as depicted below; a +drbd_peer_device object sits at each intersection between a drbd_device and a +drbd_connection: + + /--------------+---------------+.....+---------------\ + | resource | device | | device | + +--------------+---------------+.....+---------------+ + | connection | peer_device | | peer_device | + +--------------+---------------+.....+---------------+ + : : : : : + : : : : : + +--------------+---------------+.....+---------------+ + | connection | peer_device | | peer_device | + \--------------+---------------+.....+---------------/ + +In this table, horizontally, devices can be accessed from resources by their +volume number. Likewise, peer_devices can be accessed from connections by +their volume number. Objects in the vertical direction are connected by double +linked lists. There are back pointers from peer_devices to their connections a +devices, and from connections and devices to their resource. + +All resources are in the drbd_resources double-linked list. In addition, all +devices can be accessed by their minor device number via the drbd_devices idr. + +The drbd_resource, drbd_connection, and drbd_device objects are reference +counted. The peer_device objects only serve to establish the links between +devices and connections; their lifetime is determined by the lifetime of the +device and connection which they reference. diff --git a/Documentation/blockdev/drbd/disk-states-8.dot b/Documentation/blockdev/drbd/disk-states-8.dot new file mode 100644 index 000000000..d06cfb46f --- /dev/null +++ b/Documentation/blockdev/drbd/disk-states-8.dot @@ -0,0 +1,16 @@ +digraph disk_states { + Diskless -> Inconsistent [ label = "ioctl_set_disk()" ] + Diskless -> Consistent [ label = "ioctl_set_disk()" ] + Diskless -> Outdated [ label = "ioctl_set_disk()" ] + Consistent -> Outdated [ label = "receive_param()" ] + Consistent -> UpToDate [ label = "receive_param()" ] + Consistent -> Inconsistent [ label = "start resync" ] + Outdated -> Inconsistent [ label = "start resync" ] + UpToDate -> Inconsistent [ label = "ioctl_replicate" ] + Inconsistent -> UpToDate [ label = "resync completed" ] + Consistent -> Failed [ label = "io completion error" ] + Outdated -> Failed [ label = "io completion error" ] + UpToDate -> Failed [ label = "io completion error" ] + Inconsistent -> Failed [ label = "io completion error" ] + Failed -> Diskless [ label = "sending notify to peer" ] +} diff --git a/Documentation/blockdev/drbd/drbd-connection-state-overview.dot b/Documentation/blockdev/drbd/drbd-connection-state-overview.dot new file mode 100644 index 000000000..6d9cf0a7b --- /dev/null +++ b/Documentation/blockdev/drbd/drbd-connection-state-overview.dot @@ -0,0 +1,85 @@ +// vim: set sw=2 sts=2 : +digraph { + rankdir=BT + bgcolor=white + + node [shape=plaintext] + node [fontcolor=black] + + StandAlone [ style=filled,fillcolor=gray,label=StandAlone ] + + node [fontcolor=lightgray] + + Unconnected [ label=Unconnected ] + + CommTrouble [ shape=record, + label="{communication loss|{Timeout|BrokenPipe|NetworkFailure}}" ] + + node [fontcolor=gray] + + subgraph cluster_try_connect { + label="try to connect, handshake" + rank=max + WFConnection [ label=WFConnection ] + WFReportParams [ label=WFReportParams ] + } + + TearDown [ label=TearDown ] + + Connected [ label=Connected,style=filled,fillcolor=green,fontcolor=black ] + + node [fontcolor=lightblue] + + StartingSyncS [ label=StartingSyncS ] + StartingSyncT [ label=StartingSyncT ] + + subgraph cluster_bitmap_exchange { + node [fontcolor=red] + fontcolor=red + label="new application (WRITE?) requests blocked\lwhile bitmap is exchanged" + + WFBitMapT [ label=WFBitMapT ] + WFSyncUUID [ label=WFSyncUUID ] + WFBitMapS [ label=WFBitMapS ] + } + + node [fontcolor=blue] + + cluster_resync [ shape=record,label="{<any>resynchronisation process running\l'concurrent' application requests allowed|{{<T>PausedSyncT\nSyncTarget}|{<S>PausedSyncS\nSyncSource}}}" ] + + node [shape=box,fontcolor=black] + + // drbdadm [label="drbdadm connect"] + // handshake [label="drbd_connect()\ndrbd_do_handshake\ndrbd_sync_handshake() etc."] + // comm_error [label="communication trouble"] + + // + // edges + // -------------------------------------- + + StandAlone -> Unconnected [ label="drbdadm connect" ] + Unconnected -> StandAlone [ label="drbdadm disconnect\lor serious communication trouble" ] + Unconnected -> WFConnection [ label="receiver thread is started" ] + WFConnection -> WFReportParams [ headlabel="accept()\land/or \lconnect()\l" ] + + WFReportParams -> StandAlone [ label="during handshake\lpeers do not agree\labout something essential" ] + WFReportParams -> Connected [ label="data identical\lno sync needed",color=green,fontcolor=green ] + + WFReportParams -> WFBitMapS + WFReportParams -> WFBitMapT + WFBitMapT -> WFSyncUUID [minlen=0.1,constraint=false] + + WFBitMapS -> cluster_resync:S + WFSyncUUID -> cluster_resync:T + + edge [color=green] + cluster_resync:any -> Connected [ label="resnyc done",fontcolor=green ] + + edge [color=red] + WFReportParams -> CommTrouble + Connected -> CommTrouble + cluster_resync:any -> CommTrouble + edge [color=black] + CommTrouble -> Unconnected [label="receiver thread is stopped" ] + +} diff --git a/Documentation/blockdev/drbd/node-states-8.dot b/Documentation/blockdev/drbd/node-states-8.dot new file mode 100644 index 000000000..4a2b00c23 --- /dev/null +++ b/Documentation/blockdev/drbd/node-states-8.dot @@ -0,0 +1,14 @@ +digraph node_states { + Secondary -> Primary [ label = "ioctl_set_state()" ] + Primary -> Secondary [ label = "ioctl_set_state()" ] +} + +digraph peer_states { + Secondary -> Primary [ label = "recv state packet" ] + Primary -> Secondary [ label = "recv state packet" ] + Primary -> Unknown [ label = "connection lost" ] + Secondary -> Unknown [ label = "connection lost" ] + Unknown -> Primary [ label = "connected" ] + Unknown -> Secondary [ label = "connected" ] +} + diff --git a/Documentation/blockdev/floppy.txt b/Documentation/blockdev/floppy.txt new file mode 100644 index 000000000..e2240f5ab --- /dev/null +++ b/Documentation/blockdev/floppy.txt @@ -0,0 +1,245 @@ +This file describes the floppy driver. + +FAQ list: +========= + + A FAQ list may be found in the fdutils package (see below), and also +at <http://fdutils.linux.lu/faq.html>. + + +LILO configuration options (Thinkpad users, read this) +====================================================== + + The floppy driver is configured using the 'floppy=' option in +lilo. This option can be typed at the boot prompt, or entered in the +lilo configuration file. + + Example: If your kernel is called linux-2.6.9, type the following line +at the lilo boot prompt (if you have a thinkpad): + + linux-2.6.9 floppy=thinkpad + +You may also enter the following line in /etc/lilo.conf, in the description +of linux-2.6.9: + + append = "floppy=thinkpad" + + Several floppy related options may be given, example: + + linux-2.6.9 floppy=daring floppy=two_fdc + append = "floppy=daring floppy=two_fdc" + + If you give options both in the lilo config file and on the boot +prompt, the option strings of both places are concatenated, the boot +prompt options coming last. That's why there are also options to +restore the default behavior. + + +Module configuration options +============================ + + If you use the floppy driver as a module, use the following syntax: +modprobe floppy floppy="<options>" + +Example: + modprobe floppy floppy="omnibook messages" + + If you need certain options enabled every time you load the floppy driver, +you can put: + + options floppy floppy="omnibook messages" + +in a configuration file in /etc/modprobe.d/. + + + The floppy driver related options are: + + floppy=asus_pci + Sets the bit mask to allow only units 0 and 1. (default) + + floppy=daring + Tells the floppy driver that you have a well behaved floppy controller. + This allows more efficient and smoother operation, but may fail on + certain controllers. This may speed up certain operations. + + floppy=0,daring + Tells the floppy driver that your floppy controller should be used + with caution. + + floppy=one_fdc + Tells the floppy driver that you have only one floppy controller. + (default) + + floppy=two_fdc + floppy=<address>,two_fdc + Tells the floppy driver that you have two floppy controllers. + The second floppy controller is assumed to be at <address>. + This option is not needed if the second controller is at address + 0x370, and if you use the 'cmos' option. + + floppy=thinkpad + Tells the floppy driver that you have a Thinkpad. Thinkpads use an + inverted convention for the disk change line. + + floppy=0,thinkpad + Tells the floppy driver that you don't have a Thinkpad. + + floppy=omnibook + floppy=nodma + Tells the floppy driver not to use Dma for data transfers. + This is needed on HP Omnibooks, which don't have a workable + DMA channel for the floppy driver. This option is also useful + if you frequently get "Unable to allocate DMA memory" messages. + Indeed, dma memory needs to be continuous in physical memory, + and is thus harder to find, whereas non-dma buffers may be + allocated in virtual memory. However, I advise against this if + you have an FDC without a FIFO (8272A or 82072). 82072A and + later are OK. You also need at least a 486 to use nodma. + If you use nodma mode, I suggest you also set the FIFO + threshold to 10 or lower, in order to limit the number of data + transfer interrupts. + + If you have a FIFO-able FDC, the floppy driver automatically + falls back on non DMA mode if no DMA-able memory can be found. + If you want to avoid this, explicitly ask for 'yesdma'. + + floppy=yesdma + Tells the floppy driver that a workable DMA channel is available. + (default) + + floppy=nofifo + Disables the FIFO entirely. This is needed if you get "Bus + master arbitration error" messages from your Ethernet card (or + from other devices) while accessing the floppy. + + floppy=usefifo + Enables the FIFO. (default) + + floppy=<threshold>,fifo_depth + Sets the FIFO threshold. This is mostly relevant in DMA + mode. If this is higher, the floppy driver tolerates more + interrupt latency, but it triggers more interrupts (i.e. it + imposes more load on the rest of the system). If this is + lower, the interrupt latency should be lower too (faster + processor). The benefit of a lower threshold is less + interrupts. + + To tune the fifo threshold, switch on over/underrun messages + using 'floppycontrol --messages'. Then access a floppy + disk. If you get a huge amount of "Over/Underrun - retrying" + messages, then the fifo threshold is too low. Try with a + higher value, until you only get an occasional Over/Underrun. + It is a good idea to compile the floppy driver as a module + when doing this tuning. Indeed, it allows to try different + fifo values without rebooting the machine for each test. Note + that you need to do 'floppycontrol --messages' every time you + re-insert the module. + + Usually, tuning the fifo threshold should not be needed, as + the default (0xa) is reasonable. + + floppy=<drive>,<type>,cmos + Sets the CMOS type of <drive> to <type>. This is mandatory if + you have more than two floppy drives (only two can be + described in the physical CMOS), or if your BIOS uses + non-standard CMOS types. The CMOS types are: + + 0 - Use the value of the physical CMOS + 1 - 5 1/4 DD + 2 - 5 1/4 HD + 3 - 3 1/2 DD + 4 - 3 1/2 HD + 5 - 3 1/2 ED + 6 - 3 1/2 ED + 16 - unknown or not installed + + (Note: there are two valid types for ED drives. This is because 5 was + initially chosen to represent floppy *tapes*, and 6 for ED drives. + AMI ignored this, and used 5 for ED drives. That's why the floppy + driver handles both.) + + floppy=unexpected_interrupts + Print a warning message when an unexpected interrupt is received. + (default) + + floppy=no_unexpected_interrupts + floppy=L40SX + Don't print a message when an unexpected interrupt is received. This + is needed on IBM L40SX laptops in certain video modes. (There seems + to be an interaction between video and floppy. The unexpected + interrupts affect only performance, and can be safely ignored.) + + floppy=broken_dcl + Don't use the disk change line, but assume that the disk was + changed whenever the device node is reopened. Needed on some + boxes where the disk change line is broken or unsupported. + This should be regarded as a stopgap measure, indeed it makes + floppy operation less efficient due to unneeded cache + flushings, and slightly more unreliable. Please verify your + cable, connection and jumper settings if you have any DCL + problems. However, some older drives, and also some laptops + are known not to have a DCL. + + floppy=debug + Print debugging messages. + + floppy=messages + Print informational messages for some operations (disk change + notifications, warnings about over and underruns, and about + autodetection). + + floppy=silent_dcl_clear + Uses a less noisy way to clear the disk change line (which + doesn't involve seeks). Implied by 'daring' option. + + floppy=<nr>,irq + Sets the floppy IRQ to <nr> instead of 6. + + floppy=<nr>,dma + Sets the floppy DMA channel to <nr> instead of 2. + + floppy=slow + Use PS/2 stepping rate: + " PS/2 floppies have much slower step rates than regular floppies. + It's been recommended that take about 1/4 of the default speed + in some more extreme cases." + + +Supporting utilities and additional documentation: +================================================== + + Additional parameters of the floppy driver can be configured at +runtime. Utilities which do this can be found in the fdutils package. +This package also contains a new version of mtools which allows to +access high capacity disks (up to 1992K on a high density 3 1/2 disk!). +It also contains additional documentation about the floppy driver. + +The latest version can be found at fdutils homepage: + http://fdutils.linux.lu + +The fdutils releases can be found at: + http://fdutils.linux.lu/download.html + http://www.tux.org/pub/knaff/fdutils/ + ftp://metalab.unc.edu/pub/Linux/utils/disk-management/ + +Reporting problems about the floppy driver +========================================== + + If you have a question or a bug report about the floppy driver, mail +me at Alain.Knaff@poboxes.com . If you post to Usenet, preferably use +comp.os.linux.hardware. As the volume in these groups is rather high, +be sure to include the word "floppy" (or "FLOPPY") in the subject +line. If the reported problem happens when mounting floppy disks, be +sure to mention also the type of the filesystem in the subject line. + + Be sure to read the FAQ before mailing/posting any bug reports! + + Alain + +Changelog +========= + +10-30-2004 : Cleanup, updating, add reference to module configuration. + James Nelson <james4765@gmail.com> + +6-3-2000 : Original Document diff --git a/Documentation/blockdev/nbd.txt b/Documentation/blockdev/nbd.txt new file mode 100644 index 000000000..db242ea2b --- /dev/null +++ b/Documentation/blockdev/nbd.txt @@ -0,0 +1,31 @@ +Network Block Device (TCP version) +================================== + +1) Overview +----------- + +What is it: With this compiled in the kernel (or as a module), Linux +can use a remote server as one of its block devices. So every time +the client computer wants to read, e.g., /dev/nb0, it sends a +request over TCP to the server, which will reply with the data read. +This can be used for stations with low disk space (or even diskless) +to borrow disk space from another computer. +Unlike NFS, it is possible to put any filesystem on it, etc. + +For more information, or to download the nbd-client and nbd-server +tools, go to http://nbd.sf.net/. + +The nbd kernel module need only be installed on the client +system, as the nbd-server is completely in userspace. In fact, +the nbd-server has been successfully ported to other operating +systems, including Windows. + +A) NBD parameters +----------------- + +max_part + Number of partitions per device (default: 0). + +nbds_max + Number of block devices that should be initialized (default: 16). + diff --git a/Documentation/blockdev/paride.txt b/Documentation/blockdev/paride.txt new file mode 100644 index 000000000..ee6717e37 --- /dev/null +++ b/Documentation/blockdev/paride.txt @@ -0,0 +1,417 @@ + + Linux and parallel port IDE devices + +PARIDE v1.03 (c) 1997-8 Grant Guenther <grant@torque.net> + +1. Introduction + +Owing to the simplicity and near universality of the parallel port interface +to personal computers, many external devices such as portable hard-disk, +CD-ROM, LS-120 and tape drives use the parallel port to connect to their +host computer. While some devices (notably scanners) use ad-hoc methods +to pass commands and data through the parallel port interface, most +external devices are actually identical to an internal model, but with +a parallel-port adapter chip added in. Some of the original parallel port +adapters were little more than mechanisms for multiplexing a SCSI bus. +(The Iomega PPA-3 adapter used in the ZIP drives is an example of this +approach). Most current designs, however, take a different approach. +The adapter chip reproduces a small ISA or IDE bus in the external device +and the communication protocol provides operations for reading and writing +device registers, as well as data block transfer functions. Sometimes, +the device being addressed via the parallel cable is a standard SCSI +controller like an NCR 5380. The "ditto" family of external tape +drives use the ISA replicator to interface a floppy disk controller, +which is then connected to a floppy-tape mechanism. The vast majority +of external parallel port devices, however, are now based on standard +IDE type devices, which require no intermediate controller. If one +were to open up a parallel port CD-ROM drive, for instance, one would +find a standard ATAPI CD-ROM drive, a power supply, and a single adapter +that interconnected a standard PC parallel port cable and a standard +IDE cable. It is usually possible to exchange the CD-ROM device with +any other device using the IDE interface. + +The document describes the support in Linux for parallel port IDE +devices. It does not cover parallel port SCSI devices, "ditto" tape +drives or scanners. Many different devices are supported by the +parallel port IDE subsystem, including: + + MicroSolutions backpack CD-ROM + MicroSolutions backpack PD/CD + MicroSolutions backpack hard-drives + MicroSolutions backpack 8000t tape drive + SyQuest EZ-135, EZ-230 & SparQ drives + Avatar Shark + Imation Superdisk LS-120 + Maxell Superdisk LS-120 + FreeCom Power CD + Hewlett-Packard 5GB and 8GB tape drives + Hewlett-Packard 7100 and 7200 CD-RW drives + +as well as most of the clone and no-name products on the market. + +To support such a wide range of devices, PARIDE, the parallel port IDE +subsystem, is actually structured in three parts. There is a base +paride module which provides a registry and some common methods for +accessing the parallel ports. The second component is a set of +high-level drivers for each of the different types of supported devices: + + pd IDE disk + pcd ATAPI CD-ROM + pf ATAPI disk + pt ATAPI tape + pg ATAPI generic + +(Currently, the pg driver is only used with CD-R drives). + +The high-level drivers function according to the relevant standards. +The third component of PARIDE is a set of low-level protocol drivers +for each of the parallel port IDE adapter chips. Thanks to the interest +and encouragement of Linux users from many parts of the world, +support is available for almost all known adapter protocols: + + aten ATEN EH-100 (HK) + bpck Microsolutions backpack (US) + comm DataStor (old-type) "commuter" adapter (TW) + dstr DataStor EP-2000 (TW) + epat Shuttle EPAT (UK) + epia Shuttle EPIA (UK) + fit2 FIT TD-2000 (US) + fit3 FIT TD-3000 (US) + friq Freecom IQ cable (DE) + frpw Freecom Power (DE) + kbic KingByte KBIC-951A and KBIC-971A (TW) + ktti KT Technology PHd adapter (SG) + on20 OnSpec 90c20 (US) + on26 OnSpec 90c26 (US) + + +2. Using the PARIDE subsystem + +While configuring the Linux kernel, you may choose either to build +the PARIDE drivers into your kernel, or to build them as modules. + +In either case, you will need to select "Parallel port IDE device support" +as well as at least one of the high-level drivers and at least one +of the parallel port communication protocols. If you do not know +what kind of parallel port adapter is used in your drive, you could +begin by checking the file names and any text files on your DOS +installation floppy. Alternatively, you can look at the markings on +the adapter chip itself. That's usually sufficient to identify the +correct device. + +You can actually select all the protocol modules, and allow the PARIDE +subsystem to try them all for you. + +For the "brand-name" products listed above, here are the protocol +and high-level drivers that you would use: + + Manufacturer Model Driver Protocol + + MicroSolutions CD-ROM pcd bpck + MicroSolutions PD drive pf bpck + MicroSolutions hard-drive pd bpck + MicroSolutions 8000t tape pt bpck + SyQuest EZ, SparQ pd epat + Imation Superdisk pf epat + Maxell Superdisk pf friq + Avatar Shark pd epat + FreeCom CD-ROM pcd frpw + Hewlett-Packard 5GB Tape pt epat + Hewlett-Packard 7200e (CD) pcd epat + Hewlett-Packard 7200e (CD-R) pg epat + +2.1 Configuring built-in drivers + +We recommend that you get to know how the drivers work and how to +configure them as loadable modules, before attempting to compile a +kernel with the drivers built-in. + +If you built all of your PARIDE support directly into your kernel, +and you have just a single parallel port IDE device, your kernel should +locate it automatically for you. If you have more than one device, +you may need to give some command line options to your bootloader +(eg: LILO), how to do that is beyond the scope of this document. + +The high-level drivers accept a number of command line parameters, all +of which are documented in the source files in linux/drivers/block/paride. +By default, each driver will automatically try all parallel ports it +can find, and all protocol types that have been installed, until it finds +a parallel port IDE adapter. Once it finds one, the probe stops. So, +if you have more than one device, you will need to tell the drivers +how to identify them. This requires specifying the port address, the +protocol identification number and, for some devices, the drive's +chain ID. While your system is booting, a number of messages are +displayed on the console. Like all such messages, they can be +reviewed with the 'dmesg' command. Among those messages will be +some lines like: + + paride: bpck registered as protocol 0 + paride: epat registered as protocol 1 + +The numbers will always be the same until you build a new kernel with +different protocol selections. You should note these numbers as you +will need them to identify the devices. + +If you happen to be using a MicroSolutions backpack device, you will +also need to know the unit ID number for each drive. This is usually +the last two digits of the drive's serial number (but read MicroSolutions' +documentation about this). + +As an example, let's assume that you have a MicroSolutions PD/CD drive +with unit ID number 36 connected to the parallel port at 0x378, a SyQuest +EZ-135 connected to the chained port on the PD/CD drive and also an +Imation Superdisk connected to port 0x278. You could give the following +options on your boot command: + + pd.drive0=0x378,1 pf.drive0=0x278,1 pf.drive1=0x378,0,36 + +In the last option, pf.drive1 configures device /dev/pf1, the 0x378 +is the parallel port base address, the 0 is the protocol registration +number and 36 is the chain ID. + +Please note: while PARIDE will work both with and without the +PARPORT parallel port sharing system that is included by the +"Parallel port support" option, PARPORT must be included and enabled +if you want to use chains of devices on the same parallel port. + +2.2 Loading and configuring PARIDE as modules + +It is much faster and simpler to get to understand the PARIDE drivers +if you use them as loadable kernel modules. + +Note 1: using these drivers with the "kerneld" automatic module loading +system is not recommended for beginners, and is not documented here. + +Note 2: if you build PARPORT support as a loadable module, PARIDE must +also be built as loadable modules, and PARPORT must be loaded before the +PARIDE modules. + +To use PARIDE, you must begin by + + insmod paride + +this loads a base module which provides a registry for the protocols, +among other tasks. + +Then, load as many of the protocol modules as you think you might need. +As you load each module, it will register the protocols that it supports, +and print a log message to your kernel log file and your console. For +example: + + # insmod epat + paride: epat registered as protocol 0 + # insmod kbic + paride: k951 registered as protocol 1 + paride: k971 registered as protocol 2 + +Finally, you can load high-level drivers for each kind of device that +you have connected. By default, each driver will autoprobe for a single +device, but you can support up to four similar devices by giving their +individual co-ordinates when you load the driver. + +For example, if you had two no-name CD-ROM drives both using the +KingByte KBIC-951A adapter, one on port 0x378 and the other on 0x3bc +you could give the following command: + + # insmod pcd drive0=0x378,1 drive1=0x3bc,1 + +For most adapters, giving a port address and protocol number is sufficient, +but check the source files in linux/drivers/block/paride for more +information. (Hopefully someone will write some man pages one day !). + +As another example, here's what happens when PARPORT is installed, and +a SyQuest EZ-135 is attached to port 0x378: + + # insmod paride + paride: version 1.0 installed + # insmod epat + paride: epat registered as protocol 0 + # insmod pd + pd: pd version 1.0, major 45, cluster 64, nice 0 + pda: Sharing parport1 at 0x378 + pda: epat 1.0, Shuttle EPAT chip c3 at 0x378, mode 5 (EPP-32), delay 1 + pda: SyQuest EZ135A, 262144 blocks [128M], (512/16/32), removable media + pda: pda1 + +Note that the last line is the output from the generic partition table +scanner - in this case it reports that it has found a disk with one partition. + +2.3 Using a PARIDE device + +Once the drivers have been loaded, you can access PARIDE devices in the +same way as their traditional counterparts. You will probably need to +create the device "special files". Here is a simple script that you can +cut to a file and execute: + +#!/bin/bash +# +# mkd -- a script to create the device special files for the PARIDE subsystem +# +function mkdev { + mknod $1 $2 $3 $4 ; chmod 0660 $1 ; chown root:disk $1 +} +# +function pd { + D=$( printf \\$( printf "x%03x" $[ $1 + 97 ] ) ) + mkdev pd$D b 45 $[ $1 * 16 ] + for P in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 + do mkdev pd$D$P b 45 $[ $1 * 16 + $P ] + done +} +# +cd /dev +# +for u in 0 1 2 3 ; do pd $u ; done +for u in 0 1 2 3 ; do mkdev pcd$u b 46 $u ; done +for u in 0 1 2 3 ; do mkdev pf$u b 47 $u ; done +for u in 0 1 2 3 ; do mkdev pt$u c 96 $u ; done +for u in 0 1 2 3 ; do mkdev npt$u c 96 $[ $u + 128 ] ; done +for u in 0 1 2 3 ; do mkdev pg$u c 97 $u ; done +# +# end of mkd + +With the device files and drivers in place, you can access PARIDE devices +like any other Linux device. For example, to mount a CD-ROM in pcd0, use: + + mount /dev/pcd0 /cdrom + +If you have a fresh Avatar Shark cartridge, and the drive is pda, you +might do something like: + + fdisk /dev/pda -- make a new partition table with + partition 1 of type 83 + + mke2fs /dev/pda1 -- to build the file system + + mkdir /shark -- make a place to mount the disk + + mount /dev/pda1 /shark + +Devices like the Imation superdisk work in the same way, except that +they do not have a partition table. For example to make a 120MB +floppy that you could share with a DOS system: + + mkdosfs /dev/pf0 + mount /dev/pf0 /mnt + + +2.4 The pf driver + +The pf driver is intended for use with parallel port ATAPI disk +devices. The most common devices in this category are PD drives +and LS-120 drives. Traditionally, media for these devices are not +partitioned. Consequently, the pf driver does not support partitioned +media. This may be changed in a future version of the driver. + +2.5 Using the pt driver + +The pt driver for parallel port ATAPI tape drives is a minimal driver. +It does not yet support many of the standard tape ioctl operations. +For best performance, a block size of 32KB should be used. You will +probably want to set the parallel port delay to 0, if you can. + +2.6 Using the pg driver + +The pg driver can be used in conjunction with the cdrecord program +to create CD-ROMs. Please get cdrecord version 1.6.1 or later +from ftp://ftp.fokus.gmd.de/pub/unix/cdrecord/ . To record CD-R media +your parallel port should ideally be set to EPP mode, and the "port delay" +should be set to 0. With those settings it is possible to record at 2x +speed without any buffer underruns. If you cannot get the driver to work +in EPP mode, try to use "bidirectional" or "PS/2" mode and 1x speeds only. + + +3. Troubleshooting + +3.1 Use EPP mode if you can + +The most common problems that people report with the PARIDE drivers +concern the parallel port CMOS settings. At this time, none of the +PARIDE protocol modules support ECP mode, or any ECP combination modes. +If you are able to do so, please set your parallel port into EPP mode +using your CMOS setup procedure. + +3.2 Check the port delay + +Some parallel ports cannot reliably transfer data at full speed. To +offset the errors, the PARIDE protocol modules introduce a "port +delay" between each access to the i/o ports. Each protocol sets +a default value for this delay. In most cases, the user can override +the default and set it to 0 - resulting in somewhat higher transfer +rates. In some rare cases (especially with older 486 systems) the +default delays are not long enough. if you experience corrupt data +transfers, or unexpected failures, you may wish to increase the +port delay. The delay can be programmed using the "driveN" parameters +to each of the high-level drivers. Please see the notes above, or +read the comments at the beginning of the driver source files in +linux/drivers/block/paride. + +3.3 Some drives need a printer reset + +There appear to be a number of "noname" external drives on the market +that do not always power up correctly. We have noticed this with some +drives based on OnSpec and older Freecom adapters. In these rare cases, +the adapter can often be reinitialised by issuing a "printer reset" on +the parallel port. As the reset operation is potentially disruptive in +multiple device environments, the PARIDE drivers will not do it +automatically. You can however, force a printer reset by doing: + + insmod lp reset=1 + rmmod lp + +If you have one of these marginal cases, you should probably build +your paride drivers as modules, and arrange to do the printer reset +before loading the PARIDE drivers. + +3.4 Use the verbose option and dmesg if you need help + +While a lot of testing has gone into these drivers to make them work +as smoothly as possible, problems will arise. If you do have problems, +please check all the obvious things first: does the drive work in +DOS with the manufacturer's drivers ? If that doesn't yield any useful +clues, then please make sure that only one drive is hooked to your system, +and that either (a) PARPORT is enabled or (b) no other device driver +is using your parallel port (check in /proc/ioports). Then, load the +appropriate drivers (you can load several protocol modules if you want) +as in: + + # insmod paride + # insmod epat + # insmod bpck + # insmod kbic + ... + # insmod pd verbose=1 + +(using the correct driver for the type of device you have, of course). +The verbose=1 parameter will cause the drivers to log a trace of their +activity as they attempt to locate your drive. + +Use 'dmesg' to capture a log of all the PARIDE messages (any messages +beginning with paride:, a protocol module's name or a driver's name) and +include that with your bug report. You can submit a bug report in one +of two ways. Either send it directly to the author of the PARIDE suite, +by e-mail to grant@torque.net, or join the linux-parport mailing list +and post your report there. + +3.5 For more information or help + +You can join the linux-parport mailing list by sending a mail message +to + linux-parport-request@torque.net + +with the single word + + subscribe + +in the body of the mail message (not in the subject line). Please be +sure that your mail program is correctly set up when you do this, as +the list manager is a robot that will subscribe you using the reply +address in your mail headers. REMOVE any anti-spam gimmicks you may +have in your mail headers, when sending mail to the list server. + +You might also find some useful information on the linux-parport +web pages (although they are not always up to date) at + + http://web.archive.org/web/*/http://www.torque.net/parport/ + + diff --git a/Documentation/blockdev/ramdisk.txt b/Documentation/blockdev/ramdisk.txt new file mode 100644 index 000000000..501e12e03 --- /dev/null +++ b/Documentation/blockdev/ramdisk.txt @@ -0,0 +1,174 @@ +Using the RAM disk block device with Linux +------------------------------------------ + +Contents: + + 1) Overview + 2) Kernel Command Line Parameters + 3) Using "rdev -r" + 4) An Example of Creating a Compressed RAM Disk + + +1) Overview +----------- + +The RAM disk driver is a way to use main system memory as a block device. It +is required for initrd, an initial filesystem used if you need to load modules +in order to access the root filesystem (see Documentation/admin-guide/initrd.rst). It can +also be used for a temporary filesystem for crypto work, since the contents +are erased on reboot. + +The RAM disk dynamically grows as more space is required. It does this by using +RAM from the buffer cache. The driver marks the buffers it is using as dirty +so that the VM subsystem does not try to reclaim them later. + +The RAM disk supports up to 16 RAM disks by default, and can be reconfigured +to support an unlimited number of RAM disks (at your own risk). Just change +the configuration symbol BLK_DEV_RAM_COUNT in the Block drivers config menu +and (re)build the kernel. + +To use RAM disk support with your system, run './MAKEDEV ram' from the /dev +directory. RAM disks are all major number 1, and start with minor number 0 +for /dev/ram0, etc. If used, modern kernels use /dev/ram0 for an initrd. + +The new RAM disk also has the ability to load compressed RAM disk images, +allowing one to squeeze more programs onto an average installation or +rescue floppy disk. + + +2) Parameters +--------------------------------- + +2a) Kernel Command Line Parameters + + ramdisk_size=N + ============== + +This parameter tells the RAM disk driver to set up RAM disks of N k size. The +default is 4096 (4 MB). + +2b) Module parameters + + rd_nr + ===== + /dev/ramX devices created. + + max_part + ======== + Maximum partition number. + + rd_size + ======= + See ramdisk_size. + +3) Using "rdev -r" +------------------ + +The usage of the word (two bytes) that "rdev -r" sets in the kernel image is +as follows. The low 11 bits (0 -> 10) specify an offset (in 1 k blocks) of up +to 2 MB (2^11) of where to find the RAM disk (this used to be the size). Bit +14 indicates that a RAM disk is to be loaded, and bit 15 indicates whether a +prompt/wait sequence is to be given before trying to read the RAM disk. Since +the RAM disk dynamically grows as data is being written into it, a size field +is not required. Bits 11 to 13 are not currently used and may as well be zero. +These numbers are no magical secrets, as seen below: + +./arch/x86/kernel/setup.c:#define RAMDISK_IMAGE_START_MASK 0x07FF +./arch/x86/kernel/setup.c:#define RAMDISK_PROMPT_FLAG 0x8000 +./arch/x86/kernel/setup.c:#define RAMDISK_LOAD_FLAG 0x4000 + +Consider a typical two floppy disk setup, where you will have the +kernel on disk one, and have already put a RAM disk image onto disk #2. + +Hence you want to set bits 0 to 13 as 0, meaning that your RAM disk +starts at an offset of 0 kB from the beginning of the floppy. +The command line equivalent is: "ramdisk_start=0" + +You want bit 14 as one, indicating that a RAM disk is to be loaded. +The command line equivalent is: "load_ramdisk=1" + +You want bit 15 as one, indicating that you want a prompt/keypress +sequence so that you have a chance to switch floppy disks. +The command line equivalent is: "prompt_ramdisk=1" + +Putting that together gives 2^15 + 2^14 + 0 = 49152 for an rdev word. +So to create disk one of the set, you would do: + + /usr/src/linux# cat arch/x86/boot/zImage > /dev/fd0 + /usr/src/linux# rdev /dev/fd0 /dev/fd0 + /usr/src/linux# rdev -r /dev/fd0 49152 + +If you make a boot disk that has LILO, then for the above, you would use: + append = "ramdisk_start=0 load_ramdisk=1 prompt_ramdisk=1" +Since the default start = 0 and the default prompt = 1, you could use: + append = "load_ramdisk=1" + + +4) An Example of Creating a Compressed RAM Disk +---------------------------------------------- + +To create a RAM disk image, you will need a spare block device to +construct it on. This can be the RAM disk device itself, or an +unused disk partition (such as an unmounted swap partition). For this +example, we will use the RAM disk device, "/dev/ram0". + +Note: This technique should not be done on a machine with less than 8 MB +of RAM. If using a spare disk partition instead of /dev/ram0, then this +restriction does not apply. + +a) Decide on the RAM disk size that you want. Say 2 MB for this example. + Create it by writing to the RAM disk device. (This step is not currently + required, but may be in the future.) It is wise to zero out the + area (esp. for disks) so that maximal compression is achieved for + the unused blocks of the image that you are about to create. + + dd if=/dev/zero of=/dev/ram0 bs=1k count=2048 + +b) Make a filesystem on it. Say ext2fs for this example. + + mke2fs -vm0 /dev/ram0 2048 + +c) Mount it, copy the files you want to it (eg: /etc/* /dev/* ...) + and unmount it again. + +d) Compress the contents of the RAM disk. The level of compression + will be approximately 50% of the space used by the files. Unused + space on the RAM disk will compress to almost nothing. + + dd if=/dev/ram0 bs=1k count=2048 | gzip -v9 > /tmp/ram_image.gz + +e) Put the kernel onto the floppy + + dd if=zImage of=/dev/fd0 bs=1k + +f) Put the RAM disk image onto the floppy, after the kernel. Use an offset + that is slightly larger than the kernel, so that you can put another + (possibly larger) kernel onto the same floppy later without overlapping + the RAM disk image. An offset of 400 kB for kernels about 350 kB in + size would be reasonable. Make sure offset+size of ram_image.gz is + not larger than the total space on your floppy (usually 1440 kB). + + dd if=/tmp/ram_image.gz of=/dev/fd0 bs=1k seek=400 + +g) Use "rdev" to set the boot device, RAM disk offset, prompt flag, etc. + For prompt_ramdisk=1, load_ramdisk=1, ramdisk_start=400, one would + have 2^15 + 2^14 + 400 = 49552. + + rdev /dev/fd0 /dev/fd0 + rdev -r /dev/fd0 49552 + +That is it. You now have your boot/root compressed RAM disk floppy. Some +users may wish to combine steps (d) and (f) by using a pipe. + +-------------------------------------------------------------------------- + Paul Gortmaker 12/95 + +Changelog: +---------- + +10-22-04 : Updated to reflect changes in command line options, remove + obsolete references, general cleanup. + James Nelson (james4765@gmail.com) + + +12-95 : Original Document diff --git a/Documentation/blockdev/zram.txt b/Documentation/blockdev/zram.txt new file mode 100644 index 000000000..875b2b56b --- /dev/null +++ b/Documentation/blockdev/zram.txt @@ -0,0 +1,271 @@ +zram: Compressed RAM based block devices +---------------------------------------- + +* Introduction + +The zram module creates RAM based block devices named /dev/zram<id> +(<id> = 0, 1, ...). Pages written to these disks are compressed and stored +in memory itself. These disks allow very fast I/O and compression provides +good amounts of memory savings. Some of the usecases include /tmp storage, +use as swap disks, various caches under /var and maybe many more :) + +Statistics for individual zram devices are exported through sysfs nodes at +/sys/block/zram<id>/ + +* Usage + +There are several ways to configure and manage zram device(-s): +a) using zram and zram_control sysfs attributes +b) using zramctl utility, provided by util-linux (util-linux@vger.kernel.org). + +In this document we will describe only 'manual' zram configuration steps, +IOW, zram and zram_control sysfs attributes. + +In order to get a better idea about zramctl please consult util-linux +documentation, zramctl man-page or `zramctl --help'. Please be informed +that zram maintainers do not develop/maintain util-linux or zramctl, should +you have any questions please contact util-linux@vger.kernel.org + +Following shows a typical sequence of steps for using zram. + +WARNING +======= +For the sake of simplicity we skip error checking parts in most of the +examples below. However, it is your sole responsibility to handle errors. + +zram sysfs attributes always return negative values in case of errors. +The list of possible return codes: +-EBUSY -- an attempt to modify an attribute that cannot be changed once +the device has been initialised. Please reset device first; +-ENOMEM -- zram was not able to allocate enough memory to fulfil your +needs; +-EINVAL -- invalid input has been provided. + +If you use 'echo', the returned value that is changed by 'echo' utility, +and, in general case, something like: + + echo 3 > /sys/block/zram0/max_comp_streams + if [ $? -ne 0 ]; + handle_error + fi + +should suffice. + +1) Load Module: + modprobe zram num_devices=4 + This creates 4 devices: /dev/zram{0,1,2,3} + +num_devices parameter is optional and tells zram how many devices should be +pre-created. Default: 1. + +2) Set max number of compression streams +Regardless the value passed to this attribute, ZRAM will always +allocate multiple compression streams - one per online CPUs - thus +allowing several concurrent compression operations. The number of +allocated compression streams goes down when some of the CPUs +become offline. There is no single-compression-stream mode anymore, +unless you are running a UP system or has only 1 CPU online. + +To find out how many streams are currently available: + cat /sys/block/zram0/max_comp_streams + +3) Select compression algorithm +Using comp_algorithm device attribute one can see available and +currently selected (shown in square brackets) compression algorithms, +change selected compression algorithm (once the device is initialised +there is no way to change compression algorithm). + +Examples: + #show supported compression algorithms + cat /sys/block/zram0/comp_algorithm + lzo [lz4] + + #select lzo compression algorithm + echo lzo > /sys/block/zram0/comp_algorithm + +For the time being, the `comp_algorithm' content does not necessarily +show every compression algorithm supported by the kernel. We keep this +list primarily to simplify device configuration and one can configure +a new device with a compression algorithm that is not listed in +`comp_algorithm'. The thing is that, internally, ZRAM uses Crypto API +and, if some of the algorithms were built as modules, it's impossible +to list all of them using, for instance, /proc/crypto or any other +method. This, however, has an advantage of permitting the usage of +custom crypto compression modules (implementing S/W or H/W compression). + +4) Set Disksize +Set disk size by writing the value to sysfs node 'disksize'. +The value can be either in bytes or you can use mem suffixes. +Examples: + # Initialize /dev/zram0 with 50MB disksize + echo $((50*1024*1024)) > /sys/block/zram0/disksize + + # Using mem suffixes + echo 256K > /sys/block/zram0/disksize + echo 512M > /sys/block/zram0/disksize + echo 1G > /sys/block/zram0/disksize + +Note: +There is little point creating a zram of greater than twice the size of memory +since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the +size of the disk when not in use so a huge zram is wasteful. + +5) Set memory limit: Optional +Set memory limit by writing the value to sysfs node 'mem_limit'. +The value can be either in bytes or you can use mem suffixes. +In addition, you could change the value in runtime. +Examples: + # limit /dev/zram0 with 50MB memory + echo $((50*1024*1024)) > /sys/block/zram0/mem_limit + + # Using mem suffixes + echo 256K > /sys/block/zram0/mem_limit + echo 512M > /sys/block/zram0/mem_limit + echo 1G > /sys/block/zram0/mem_limit + + # To disable memory limit + echo 0 > /sys/block/zram0/mem_limit + +6) Activate: + mkswap /dev/zram0 + swapon /dev/zram0 + + mkfs.ext4 /dev/zram1 + mount /dev/zram1 /tmp + +7) Add/remove zram devices + +zram provides a control interface, which enables dynamic (on-demand) device +addition and removal. + +In order to add a new /dev/zramX device, perform read operation on hot_add +attribute. This will return either new device's device id (meaning that you +can use /dev/zram<id>) or error code. + +Example: + cat /sys/class/zram-control/hot_add + 1 + +To remove the existing /dev/zramX device (where X is a device id) +execute + echo X > /sys/class/zram-control/hot_remove + +8) Stats: +Per-device statistics are exported as various nodes under /sys/block/zram<id>/ + +A brief description of exported device attributes. For more details please +read Documentation/ABI/testing/sysfs-block-zram. + +Name access description +---- ------ ----------- +disksize RW show and set the device's disk size +initstate RO shows the initialization state of the device +reset WO trigger device reset +mem_used_max WO reset the `mem_used_max' counter (see later) +mem_limit WO specifies the maximum amount of memory ZRAM can use + to store the compressed data +max_comp_streams RW the number of possible concurrent compress operations +comp_algorithm RW show and change the compression algorithm +compact WO trigger memory compaction +debug_stat RO this file is used for zram debugging purposes +backing_dev RW set up backend storage for zram to write out + + +User space is advised to use the following files to read the device statistics. + +File /sys/block/zram<id>/stat + +Represents block layer statistics. Read Documentation/block/stat.txt for +details. + +File /sys/block/zram<id>/io_stat + +The stat file represents device's I/O statistics not accounted by block +layer and, thus, not available in zram<id>/stat file. It consists of a +single line of text and contains the following stats separated by +whitespace: + failed_reads the number of failed reads + failed_writes the number of failed writes + invalid_io the number of non-page-size-aligned I/O requests + notify_free Depending on device usage scenario it may account + a) the number of pages freed because of swap slot free + notifications or b) the number of pages freed because of + REQ_DISCARD requests sent by bio. The former ones are + sent to a swap block device when a swap slot is freed, + which implies that this disk is being used as a swap disk. + The latter ones are sent by filesystem mounted with + discard option, whenever some data blocks are getting + discarded. + +File /sys/block/zram<id>/mm_stat + +The stat file represents device's mm statistics. It consists of a single +line of text and contains the following stats separated by whitespace: + orig_data_size uncompressed size of data stored in this disk. + This excludes same-element-filled pages (same_pages) since + no memory is allocated for them. + Unit: bytes + compr_data_size compressed size of data stored in this disk + mem_used_total the amount of memory allocated for this disk. This + includes allocator fragmentation and metadata overhead, + allocated for this disk. So, allocator space efficiency + can be calculated using compr_data_size and this statistic. + Unit: bytes + mem_limit the maximum amount of memory ZRAM can use to store + the compressed data + mem_used_max the maximum amount of memory zram have consumed to + store the data + same_pages the number of same element filled pages written to this disk. + No memory is allocated for such pages. + pages_compacted the number of pages freed during compaction + huge_pages the number of incompressible pages + +9) Deactivate: + swapoff /dev/zram0 + umount /dev/zram1 + +10) Reset: + Write any positive value to 'reset' sysfs node + echo 1 > /sys/block/zram0/reset + echo 1 > /sys/block/zram1/reset + + This frees all the memory allocated for the given device and + resets the disksize to zero. You must set the disksize again + before reusing the device. + +* Optional Feature + += writeback + +With incompressible pages, there is no memory saving with zram. +Instead, with CONFIG_ZRAM_WRITEBACK, zram can write incompressible page +to backing storage rather than keeping it in memory. +User should set up backing device via /sys/block/zramX/backing_dev +before disksize setting. + += memory tracking + +With CONFIG_ZRAM_MEMORY_TRACKING, user can know information of the +zram block. It could be useful to catch cold or incompressible +pages of the process with*pagemap. +If you enable the feature, you could see block state via +/sys/kernel/debug/zram/zram0/block_state". The output is as follows, + + 300 75.033841 .wh + 301 63.806904 s.. + 302 63.806919 ..h + +First column is zram's block index. +Second column is access time since the system was booted +Third column is state of the block. +(s: same page +w: written page to backing store +h: huge page) + +First line of above example says 300th block is accessed at 75.033841sec +and the block's state is huge so it is written back to the backing +storage. It's a debugging feature so anyone shouldn't rely on it to work +properly. + +Nitin Gupta +ngupta@vflare.org |
