summaryrefslogtreecommitdiffstats
path: root/doc/cephfs/posix.rst
diff options
context:
space:
mode:
Diffstat (limited to 'doc/cephfs/posix.rst')
-rw-r--r--doc/cephfs/posix.rst104
1 files changed, 104 insertions, 0 deletions
diff --git a/doc/cephfs/posix.rst b/doc/cephfs/posix.rst
new file mode 100644
index 000000000..d79730c6f
--- /dev/null
+++ b/doc/cephfs/posix.rst
@@ -0,0 +1,104 @@
+========================
+ Differences from POSIX
+========================
+
+CephFS aims to adhere to POSIX semantics wherever possible. For
+example, in contrast to many other common network file systems like
+NFS, CephFS maintains strong cache coherency across clients. The goal
+is for processes communicating via the file system to behave the same
+when they are on different hosts as when they are on the same host.
+
+However, there are a few places where CephFS diverges from strict
+POSIX semantics for various reasons:
+
+- If a client is writing to a file and fails, its writes are not
+ necessarily atomic. That is, the client may call write(2) on a file
+ opened with O_SYNC with an 8 MB buffer and then crash and the write
+ may be only partially applied. (Almost all file systems, even local
+ file systems, have this behavior.)
+- In shared simultaneous writer situations, a write that crosses
+ object boundaries is not necessarily atomic. This means that you
+ could have writer A write "aa|aa" and writer B write "bb|bb"
+ simultaneously (where | is the object boundary), and end up with
+ "aa|bb" rather than the proper "aa|aa" or "bb|bb".
+- Sparse files propagate incorrectly to the stat(2) st_blocks field.
+ Because CephFS does not explicitly track which parts of a file are
+ allocated/written, the st_blocks field is always populated by the
+ file size divided by the block size. This will cause tools like
+ du(1) to overestimate consumed space. (The recursive size field,
+ maintained by CephFS, also includes file "holes" in its count.)
+- When a file is mapped into memory via mmap(2) on multiple hosts,
+ writes are not coherently propagated to other clients' caches. That
+ is, if a page is cached on host A, and then updated on host B, host
+ A's page is not coherently invalidated. (Shared writable mmap
+ appears to be quite rare--we have yet to hear any complaints about this
+ behavior, and implementing cache coherency properly is complex.)
+- CephFS clients present a hidden ``.snap`` directory that is used to
+ access, create, delete, and rename snapshots. Although the virtual
+ directory is excluded from readdir(2), any process that tries to
+ create a file or directory with the same name will get an error
+ code. The name of this hidden directory can be changed at mount
+ time with ``-o snapdirname=.somethingelse`` (Linux) or the config
+ option ``client_snapdir`` (libcephfs, ceph-fuse).
+- CephFS does not currently maintain the ``atime`` field. Most applications
+ do not care, though this impacts some backup and data tiering
+ applications that can move unused data to a secondary storage system.
+ You may be able to workaround this for some use cases, as CephFS does
+ support setting ``atime`` via the ``setattr`` operation.
+
+Perspective
+-----------
+
+People talk a lot about "POSIX compliance," but in reality most file
+system implementations do not strictly adhere to the spec, including
+local Linux file systems like ext4 and XFS. For example, for
+performance reasons, the atomicity requirements for reads are relaxed:
+processing reading from a file that is also being written may see torn
+results.
+
+Similarly, NFS has extremely weak consistency semantics when multiple
+clients are interacting with the same files or directories, opting
+instead for "close-to-open". In the world of network attached
+storage, where most environments use NFS, whether or not the server's
+file system is "fully POSIX" may not be relevant, and whether client
+applications notice depends on whether data is being shared between
+clients or not. NFS may also "tear" the results of concurrent writers
+as client data may not even be flushed to the server until the file is
+closed (and more generally writes will be significantly more
+time-shifted than CephFS, leading to less predictable results).
+
+Regardless, these are all similar enough to POSIX, and applications still work
+most of the time. Many other storage systems (e.g., HDFS) claim to be
+"POSIX-like" but diverge significantly from the standard by dropping support
+for things like in-place file modifications, truncate, or directory renames.
+
+Bottom line
+-----------
+
+CephFS relaxes more than local Linux kernel file systems (for example, writes
+spanning object boundaries may be torn). It relaxes strictly less
+than NFS when it comes to multiclient consistency, and generally less
+than NFS when it comes to write atomicity.
+
+In other words, when it comes to POSIX, ::
+
+ HDFS < NFS < CephFS < {XFS, ext4}
+
+
+fsync() and error reporting
+---------------------------
+
+POSIX is somewhat vague about the state of an inode after fsync reports
+an error. In general, CephFS uses the standard error-reporting
+mechanisms in the client's kernel, and therefore follows the same
+conventions as other file systems.
+
+In modern Linux kernels (v4.17 or later), writeback errors are reported
+once to every file description that is open at the time of the error. In
+addition, unreported errors that occurred before the file description was
+opened will also be returned on fsync.
+
+See `PostgreSQL's summary of fsync() error reporting across operating systems
+<https://wiki.postgresql.org/wiki/Fsync_Errors>`_ and `Matthew Wilcox's
+presentation on Linux IO error handling
+<https://www.youtube.com/watch?v=74c19hwY2oE>`_ for more information.