1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
|
MKSQUASHFS - a tool to create Squashfs filesystems
As Squashfs is a read-only filesystem, the Mksquashfs program must be used to
create populated squashfs filesystems.
SYNTAX:mksquashfs source1 source2 ... FILESYSTEM [OPTIONS] [-e list of
exclude dirs/files]
Filesystem compression options:
-b <block_size> set data block to <block_size>. Default 128 Kbytes.
Optionally a suffix of K or M can be given to specify
Kbytes or Mbytes respectively
-comp <comp> select <comp> compression
Compressors available:
gzip (default)
lzo
lz4
xz
zstd
-noI do not compress inode table
-noId do not compress the uid/gid table (implied by -noI)
-noD do not compress data blocks
-noF do not compress fragment blocks
-noX do not compress extended attributes
-no-compression do not compress any of the data or metadata. This is
equivalent to specifying -noI -noD -noF and -noX
Filesystem build options:
-tar read uncompressed tar file from standard in (stdin)
-no-strip act like tar, and do not strip leading directories
from source files
-tarstyle alternative name for -no-strip
-cpiostyle act like cpio, and read file pathnames from standard in
(stdin)
-cpiostyle0 like -cpiostyle, but filenames are null terminated. Can
be used with find -print0 action
-reproducible build filesystems that are reproducible (default)
-not-reproducible build filesystems that are not reproducible
-mkfs-time <time> set filesystem creation timestamp to <time>. <time> can
be an unsigned 32-bit int indicating seconds since the
epoch (1970-01-01) or a string value which is passed to
the "date" command to parse. Any string value which the
date command recognises can be used such as "now",
"last week", or "Wed Feb 15 21:02:39 GMT 2023"
-all-time <time> set all file timestamps to <time>. <time> can be an
unsigned 32-bit int indicating seconds since the epoch
(1970-01-01) or a string value which is passed to the
"date" command to parse. Any string value which the date
command recognises can be used such as "now", "last
week", or "Wed Feb 15 21:02:39 GMT 2023"
-root-time <time> set root directory time to <time>. <time> can be an
unsigned 32-bit int indicating seconds since the epoch
(1970-01-01) or a string value which is passed to the
"date" command to parse. Any string value which the date
command recognises can be used such as "now", "last
week", or "Wed Feb 15 21:02:39 GMT 2023"
-root-mode <mode> set root directory permissions to octal <mode>
-root-uid <value> set root directory owner to specified <value>,
<value> can be either an integer uid or user name
-root-gid <value> set root directory group to specified <value>,
<value> can be either an integer gid or group name
-all-root make all files owned by root
-force-uid <value> set all file uids to specified <value>, <value> can be
either an integer uid or user name
-force-gid <value> set all file gids to specified <value>, <value> can be
either an integer gid or group name
-pseudo-override make pseudo file uids and gids override -all-root,
-force-uid and -force-gid options
-no-exports do not make filesystem exportable via NFS (-tar default)
-exports make filesystem exportable via NFS (default)
-no-sparse do not detect sparse files
-no-tailends do not pack tail ends into fragments (default)
-tailends pack tail ends into fragments
-no-fragments do not use fragments
-no-duplicates do not perform duplicate checking
-no-hardlinks do not hardlink files, instead store duplicates
-keep-as-directory if one source directory is specified, create a root
directory containing that directory, rather than the
contents of the directory
Filesystem filter options:
-p <pseudo-definition> add pseudo file definition. The definition should
be quoted
-pf <pseudo-file> add list of pseudo file definitions from <pseudo-file>,
use - for stdin. Pseudo file definitions should not be
quoted
-sort <sort_file> sort files according to priorities in <sort_file>. One
file or dir with priority per line. Priority -32768 to
32767, default priority 0
-ef <exclude_file> list of exclude dirs/files. One per line
-wildcards allow extended shell wildcards (globbing) to be used in
exclude dirs/files
-regex allow POSIX regular expressions to be used in exclude
dirs/files
-max-depth <levels> descend at most <levels> of directories when scanning
filesystem
-one-file-system do not cross filesystem boundaries. If a directory
crosses the boundary, create an empty directory for
each mount point. If a file crosses the boundary
ignore it
-one-file-system-x do not cross filesystem boundaries. Like
-one-file-system option except directories are also
ignored if they cross the boundary
Filesystem extended attribute (xattrs) options:
-no-xattrs do not store extended attributes
-xattrs store extended attributes (default)
-xattrs-exclude <regex> exclude any xattr names matching <regex>. <regex> is a
POSIX regular expression, e.g. -xattrs-exclude '^user.'
excludes xattrs from the user namespace
-xattrs-include <regex> include any xattr names matching <regex>. <regex> is a
POSIX regular expression, e.g. -xattrs-include '^user.'
includes xattrs from the user namespace
-xattrs-add <name=val> add the xattr <name> with <val> to files. If an
user xattr it will be added to regular files and
directories (see man 7 xattr). Otherwise it will be
added to all files. <val> by default will be treated as
binary (i.e. an uninterpreted byte sequence), but it can
be prefixed with 0s, where it will be treated as base64
encoded, or prefixed with 0x, where val will be treated
as hexidecimal. Additionally it can be prefixed with
0t where this encoding is similar to binary encoding,
except backslashes are specially treated, and a
backslash followed by 3 octal digits can be used to
encode any ASCII character, which obviously can be used
to encode control codes. The option can be repeated
multiple times to add multiple xattrs
Mksquashfs runtime options:
-version print version, licence and copyright message
-exit-on-error treat normally ignored errors as fatal
-quiet no verbose output
-info print files written to filesystem
-no-progress do not display the progress bar
-progress display progress bar when using the -info option
-percentage display a percentage rather than the full progress bar.
Can be used with dialog --gauge etc.
-throttle <percentage> throttle the I/O input rate by the given percentage.
This can be used to reduce the I/O and CPU consumption
of Mksquashfs
-limit <percentage> limit the I/O input rate to the given percentage.
This can be used to reduce the I/O and CPU consumption
of Mksquashfs (alternative to -throttle)
-processors <number> use <number> processors. By default will use number of
processors available
-mem <size> use <size> physical memory for caches. Use K, M or G to
specify Kbytes, Mbytes or Gbytes respectively
-mem-percent <percent> use <percent> physical memory for caches. Default 25%
-mem-default print default memory usage in Mbytes
Filesystem append options:
-noappend do not append to existing filesystem
-root-becomes <name> when appending source files/directories, make the
original root become a subdirectory in the new root
called <name>, rather than adding the new source items
to the original root
-no-recovery do not generate a recovery file
-recovery-path <name> use <name> as the directory to store the recovery file
-recover <name> recover filesystem data using recovery file <name>
Filesystem actions options:
-action <action@expr> evaluate <expr> on every file, and execute <action>
if it returns TRUE
-log-action <act@expr> as above, but log expression evaluation results and
actions performed
-true-action <act@expr> as above, but only log expressions which return TRUE
-false-action <act@exp> as above, but only log expressions which return FALSE
-action-file <file> as action, but read actions from <file>
-log-action-file <file> as -log-action, but read actions from <file>
-true-action-file <f> as -true-action, but read actions from <f>
-false-action-file <f> as -false-action, but read actions from <f>
Tar file only options:
-default-mode <mode> tar files often do not store permissions for
intermediate directories. This option sets the default
directory permissions to octal <mode>, rather than 0755.
This also sets the root inode mode
-default-uid <uid> tar files often do not store uids for intermediate
directories. This option sets the default directory
owner to <uid>, rather than the user running Mksquashfs.
This also sets the root inode uid
-default-gid <gid> tar files often do not store gids for intermediate
directories. This option sets the default directory
group to <gid>, rather than the group of the user
running Mksquashfs. This also sets the root inode gid
-ignore-zeros allow tar files to be concatenated together and fed to
Mksquashfs. Normally a tarfile has two consecutive 512
byte blocks filled with zeros which means EOF and
Mksquashfs will stop reading after the first tar file on
encountering them. This option makes Mksquashfs ignore
the zero filled blocks
Expert options (these may make the filesystem unmountable):
-nopad do not pad filesystem to a multiple of 4K
-offset <offset> skip <offset> bytes at the beginning of FILESYSTEM.
Optionally a suffix of K, M or G can be given to specify
Kbytes, Mbytes or Gbytes respectively.
Default 0 bytes
-o <offset> synonym for -offset
Miscellaneous options:
-fstime <time> alternative name for -mkfs-time
-always-use-fragments alternative name for -tailends
-root-owned alternative name for -all-root
-noInodeCompression alternative name for -noI
-noIdTableCompression alternative name for -noId
-noDataCompression alternative name for -noD
-noFragmentCompression alternative name for -noF
-noXattrCompression alternative name for -noX
-help output this options text to stdout
-h output this options text to stdout
-Xhelp print compressor options for selected compressor
Pseudo file definition format:
"filename d mode uid gid" create a directory
"filename m mode uid gid" modify filename
"filename b mode uid gid major minor" create a block device
"filename c mode uid gid major minor" create a character device
"filename f mode uid gid command" create file from stdout of command
"filename s mode uid gid symlink" create a symbolic link
"filename i mode uid gid [s|f]" create a socket (s) or FIFO (f)
"filename x name=val" create an extended attribute
"filename l linkname" create a hard-link to linkname
"filename L pseudo_filename" same, but link to pseudo file
"filename D time mode uid gid" create a directory with timestamp time
"filename M time mode uid gid" modify a file with timestamp time
"filename B time mode uid gid major minor"
create block device with timestamp time
"filename C time mode uid gid major minor"
create char device with timestamp time
"filename F time mode uid gid command" create file with timestamp time
"filename S time mode uid gid symlink" create symlink with timestamp time
"filename I time mode uid gid [s|f]" create socket/fifo with timestamp time
Source1 source2 ... are the source directories/files containing the
files/directories that will form the squashfs filesystem. If a single
directory is specified (i.e. mksquashfs source image.sqfs) the squashfs
filesystem will consist of that directory, with the top-level root
directory containing the contents.
If multiple source directories or files are specified, Mksquashfs will merge
the specified sources into a single filesystem, with the root directory
containing each of the source files/directories. The name of each directory
entry will be the basename of the source path. If more than one source
entry maps to the same name, the conflicts are named xxx_1, xxx_2, etc. where
xxx is the original name.
To make this clear, take two example directories. Source directory
"/home/phillip/test" contains "file1", "file2" and "dir1".
Source directory "goodies" contains "goodies1", "goodies2" and "goodies3".
usage example 1:
% mksquashfs /home/phillip/test image.sqfs
This will generate a squashfs filesystem with root entries
"file1", "file2" and "dir1".
example 2:
% mksquashfs /home/phillip/test goodies image.sqfs
This will create a squashfs filesystem with the root containing
entries "test" and "goodies" corresponding to the source
directories "/home/phillip/test" and "goodies".
example 3:
% mksquashfs /home/phillip/test goodies test image.sqfs
This is the same as the previous example, except a third
source directory "test" has been specified. This conflicts
with the first directory named "test" and will be renamed "test_1".
Multiple sources allow filesystems to be generated without needing to
copy all source files into a common directory. This simplifies creating
filesystems.
The -keep-as-directory option can be used when only one source directory
is specified, and you wish the root to contain that directory, rather than
the contents of the directory. For example:
example 4:
% mksquashfs /home/phillip/test image.sqfs -keep-as-directory
This is the same as example 1, except for -keep-as-directory.
This will generate a root directory containing directory "test",
rather than the "test" directory contents "file1", "file2" and "dir1".
If you want the full path to retained (like TAR behaviour), you can specify the
-no-strip option.
example 5:
% mksquashfs /home/phillip/test image.sqfs -no-strip
This will make a filesystem with "home", "home/phillip" and "home/phillip/test"
directories, see section 4 for more details.
The Dest argument is the destination where the squashfs filesystem will be
written. This can either be a conventional file or a block device. If the file
doesn't exist it will be created, if it does exist and a squashfs
filesystem exists on it, Mksquashfs will append. The -noappend option will
write a new filesystem irrespective of whether an existing filesystem is
present. See section 12 for more details about appending.
1. CHANGING COMPRESSION ALGORITHM AND COMPRESSION SPECIFIC OPTIONS
------------------------------------------------------------------
By default Mksquashfs will compress using the GZIP compression algorithm. This
algorithm offers a good trade-off between compression ratio, and memory and time
taken to decompress.
Squashfs also supports LZ4, LZO, XZ and ZSTD compression. LZO offers worse
compression ratio than GZIP, but is faster to decompress. XZ offers better
compression ratio than GZIP, but at the expense of greater memory and time
to decompress (and significantly more time to compress). LZ4 is similar
to LZO. ZSTD has been developed by Facebook, and aims to compress well and
be fast to decompress. You should experiment with the compressors to
see which is best for you.
If you're not building the squashfs-tools and kernel from source, then the tools
and kernel may or may not have been built with support for LZ4, LZO, XZ or ZSTD
compression. The compression algorithms supported by the build of Mksquashfs
can be found by typing mksquashfs -help.
The full list of compressors available and their compression specific options
are:
Compressors available and compressor specific options:
gzip (default)
-Xcompression-level <compression-level>
<compression-level> should be 1 .. 9 (default 9)
-Xwindow-size <window-size>
<window-size> should be 8 .. 15 (default 15)
-Xstrategy strategy1,strategy2,...,strategyN
Compress using strategy1,strategy2,...,strategyN in turn
and choose the best compression.
Available strategies: default, filtered, huffman_only,
run_length_encoded and fixed
lzo
-Xalgorithm <algorithm>
Where <algorithm> is one of:
lzo1x_1
lzo1x_1_11
lzo1x_1_12
lzo1x_1_15
lzo1x_999 (default)
-Xcompression-level <compression-level>
<compression-level> should be 1 .. 9 (default 8)
Only applies to lzo1x_999 algorithm
lz4
-Xhc
Compress using LZ4 High Compression
xz
-Xbcj filter1,filter2,...,filterN
Compress using filter1,filter2,...,filterN in turn
(in addition to no filter), and choose the best compression.
Available filters: x86, arm, armthumb, powerpc, sparc, ia64
-Xdict-size <dict-size>
Use <dict-size> as the XZ dictionary size. The dictionary size
can be specified as a percentage of the block size, or as an
absolute value. The dictionary size must be less than or equal
to the block size and 8192 bytes or larger. It must also be
storable in the xz header as either 2^n or as 2^n+2^(n+1).
Example dict-sizes are 75%, 50%, 37.5%, 25%, or 32K, 16K, 8K
etc.
zstd
-Xcompression-level <compression-level>
<compression-level> should be 1 .. 22 (default 15)
If the compressor offers compression specific options (all the compressors now
have compression specific options except the deprecated lzma1 compressor)
then these options are also displayed (.i.e. in the above XZ is shown with two
compression specific options). The compression specific options are, obviously,
specific to the compressor in question, and the compressor documentation and
web sites should be consulted to understand their behaviour. In general
the Mksquashfs compression defaults for each compressor are optimised to
give the best performance for each compressor, where what constitutes
best depends on the compressor. For GZIP/XZ best means highest compression,
for LZO/LZ4 best means a tradeoff between compression and (de)-compression
overhead (LZO/LZ4 by definition are intended for weaker processors).
2. REDUCING CPU AND I/O USAGE
-----------------------------
By default Mksquashfs will use all the CPUs available to compress and create the
filesystem, and will read from the source files and write to the output
filesystem as fast as possible. This maximises both CPU usage and I/O.
Sometimes you don't want Mksquashfs to use all CPU and I/O bandwidth. For those
cases Mksquashfs supports two complementary options, -processors and -throttle.
The -processors option can be used to reduce the number of parallel compression
threads used by Mksquashfs. Reducing this to 1 will create the minimum number
of threads, and this will reduce CPU usage, and that in turn will reduce I/O
(because CPUs are normally the bottleneck).
The -throttle option reduces the speed Mksquashfs reads from the source files.
The value is a percentage (obviously from 1 - 100), and 50 will reduce the
read rate by half (the read thread will spend half its time idling), and 75
by three quarters. Reducing the speed of I/O will also reduce the CPU
usage as there is insufficient data rate to use all cores.
Which option should you use? Both will effectively reduce CPU and I/O in
normal cases where intensive use is being made of both I/O and CPUs. But
in edge cases there can be an imbalance where reducing one has no effect, or
it can't be reduced any further. For instance when there is only 1 or 2 cores
available, setting -processors to the minimum of 1 may still use too much
CPU. Additionally if your input source is slow Mksquashfs may still max it out
with -processors set to the minimum of 1. In this case you can use -throttle
in addition to -processors or on its own.
3. CHANGING GLOBAL COMPRESSION DEFAULTS USED IN MKSQUASHFS
----------------------------------------------------------
There are a large number of options that can be used to control the
compression in Mksquashfs. By and large the defaults are the most
optimum settings and should rarely need to be changed.
Note, this does not apply to the block size, increasing the block size from the
default of 128 Kbytes will increase compression (especially for the XZ and ZSTD
compressors) and should increase I/O performance too. However, a block size of
greater than 128 Kbytes may increase latency in certain cases (where the
filesystem contains lots of fragments, and no locality of reference is
observed). For this reason the block size default is configured to the less
optimal 128 Kbytes. Users should experiment with 256 Kbyte sizes or above.
The -b option allows the block size to be selected, both "K" and "M" postfixes
are supported, this can be either 4K, 8K, 16K, 32K, 64K, 128K, 256K, 512K or
1M bytes.
The -noI, -noD, -noF and -noX options can be used to force Mksquashfs to not
compress inodes/directories, data, fragments and extended attributes
respectively. Giving all options generates an uncompressed filesystem.
The -no-fragments option tells Mksquashfs to not generate fragment blocks. A
fragment block consists of multiple small files (all smaller than the block
size) packed and compressed together. This produces much better compression
than storing and compressing these files separately. It also typically
improves I/O as multiple files in the same directory are read at the same time.
You don't want to enable this option unless you fully understand the effects.
The -tailends option tells Mksquashfs to always generate fragments for files
irrespective of the file length. By default only small files less than the data
block size are packed into fragment blocks. The tail ends of files which do not
fit fully into a block, are NOT by default packed into fragments. This is a
legacy setting when hard disks were mechanical, and had slow seek times. In
general setting this option will gain a little more compression, without
affecting I/O performance.
The -no-duplicates option tells Mksquashfs to not check the files being added to
the filesystem for duplicates. This can result in quicker filesystem
generation although obviously compression will suffer badly if there is a lot
of duplicates.
4. TAR STYLE HANDLING OF SOURCE PATHNAMES IN MKSQUASHFS
-------------------------------------------------------
Mksquashfs has always stripped the leading directories of any source pathnames
given on the command line.
For example given the command line
% mksquashfs dir-a/dir-b/dir-c/file1 dir-A/dir-B/file2 sqfs
Mksquashfs will strip the leading directories, and put file1 and file2 into
the same root directory. If file1 and file2 are directories it will place the
directories into the same root directory, but, if only one directory is
specified, it will put the contents of that directory into the root directory.
Obviously, for a lot of the time this is what you want. But, if it isn't what
you want it can be quite difficult to get Mksquashfs to do something different.
A lot of people don't like this, and would prefer Mksquashfs acted like "tar",
which does not strip leading directories. This allows you to create a directory
hierarchy from the pathnames of the supplied files. In the above example, the
tar archive would contain the pathnames "dir-a/dir-b/dir-c/file1" and
"dir-A/dir-B/file2".
Mksquashfs will act like tar if given the option -no-strip, or -tarstyle.
5. CPIO STYLE HANDLING OF SOURCE PATHNAMES IN MKSQUASHFS
--------------------------------------------------------
Mksquashfs allows you to pipe the set of files to be added to the filesystem to
standard in (stdin), if the -cpiostyle option is given.
As with cpio, directories are not recursively scanned and their contents added.
Evey file to be added to the filesystem must be explicitly specified.
Typically the list of files to be added will be produced via find, or a similar
utility.
For example
% find /home/phillip/squashfs-tools | mksquashfs - img.sqfs -cpiostyle
Will create an image containing everything in squashfs-tools and its
sub-directories. Note, "-" is given as the source pathname in Mksquashfs, and
indicates no commmand line sources.
The following will add just the files ending in .c, .h and .o.
% find /home/phillip/squashfs-tools -name "*.[cho]" | mksquashfs - img.sqfs -cpiostyle
6. SPECIFYING THE UIDs/GIDs USED IN THE FILESYSTEM
--------------------------------------------------
By default files in the generated filesystem inherit the UID and GID ownership
of the original file. However, Mksquashfs provides a number of options which
can be used to override the ownership.
The -all-root option forces all file uids/gids in the generated Squashfs
filesystem to be root. This allows root owned filesystems to be built without
root access on the host machine.
The "-force-uid uid" option forces all files in the generated Squashfs
filesystem to be owned by the specified uid. The uid can be specified either by
name (i.e. "root") or by number.
The "-force-gid gid" option forces all files in the generated Squashfs
filesystem to be group owned by the specified gid. The gid can be specified
either by name (i.e. "root") or by number.
7. EXCLUDING FILES FROM THE FILESYSTEM
--------------------------------------
The -e and -ef options allow files/directories to be specified which are
excluded from the output filesystem. The -e option takes the exclude
files/directories from the command line, the -ef option takes the
exlude files/directories from the specified exclude file, one file/directory
per line.
Two styles of exclude file matching are supported: basic exclude matching, and
extended wildcard matching. Basic exclude matching is a legacy feature
retained for backwards compatibility with earlier versions of Mksquashfs.
Extended wildcard matching should be used in preference.
7.1 BASIC EXCLUDE MATCHING
--------------------------
Each exclude file is treated as an exact match of a file/directory in
the source directories. If an exclude file/directory is absolute (i.e.
prefixed with /, ../, or ./) the entry is treated as absolute, however, if an
exclude file/directory is relative, it is treated as being relative to each of
the sources in turn, i.e.
% mksquashfs /tmp/source1 source2 output_fs -e ex1 /tmp/source1/ex2 out/ex3
Will generate exclude files /tmp/source1/ex2, /tmp/source1/ex1, source2/ex1,
/tmp/source1/out/ex3 and source2/out/ex3.
7.2 EXTENDED EXCLUDE FILE HANDLING
----------------------------------
Extended exclude file matching treats each exclude file as a wildcard or
regex expression. To enable wildcard matching specify the -wildcards
option, and to enable regex matching specify the -regex option. In most
cases the -wildcards option should be used rather than -regex because wildcard
matching behaviour is significantly easier to understand!
In addition to wildcards/regex expressions, exclude files can be "anchored" or
"non-anchored". An anchored exclude is one which matches from the root of the
directory and nowhere else, a non-anchored exclude matches anywhere. For
example given the directory hierarchy "a/b/c/a/b", the anchored exclude
"a/b" will match "a/b" at the root of the directory hierarchy, but
it will not match the "/a/b" sub-directory within directory "c", whereas a
non-anchored exclude would.
A couple of examples should make this clearer.
Anchored excludes
1. mksquashfs example image.sqsh -wildcards -e 'test/*.gz'
Exclude all files matching "*.gz" in the top level directory "test".
2. mksquashfs example image.sqsh -wildcards -e '*/[Tt]est/example*'
Exclude all files beginning with "example" inside directories called
"Test" or "test", that occur inside any top level directory.
Using extended wildcards, negative matching is also possible.
3. mksquashfs example image.sqsh -wildcards -e 'test/!(*data*).gz'
Exclude all files matching "*.gz" in top level directory "test",
except those with "data" in the name.
Non-anchored excludes
By default excludes match from the top level directory, but it is
often useful to exclude a file matching anywhere in the source directories.
For this non-anchored excludes can be used, specified by pre-fixing the
exclude with "...".
Examples:
1. mksquashfs example image.sqsh -wildcards -e '... *.gz'
Exclude files matching "*.gz" anywhere in the source directories.
For example this will match "example.gz", "test/example.gz", and
"test/test/example.gz".
2. mksquashfs example image.sqsh -wildcards -e '... [Tt]est/*.gz'
Exclude files matching "*.gz" inside directories called "Test" or
"test" that occur anywhere in the source directories.
Again, using extended wildcards, negative matching is also possible.
3. mksquashfs example image.sqsh -wildcards -e '... !(*data*).gz'
Exclude all files matching "*.gz" anywhere in the source directories,
except those with "data" in the name.
8. FILTERING AND ADDING EXTENDED ATTRIBUTES (XATTRs)
----------------------------------------------------
Mksquashfs has a number of options which allow extended attributes (xattrs) to
be filtered from the source files or added to the created Squashfs filesystem.
The -no-xattrs option removes any extended attributes which may exist in the
source files, and creates a filesystem without any extended attributes.
The -xattrs-exclude option specifies a regular expression (regex), which
removes any extended attribute that matches the regular expression from all
files. For example the regex '^user.' will remove all User extended attributes.
The -xattrs-include option instead specifies a regular expression (regex)
which includes any extended attribute that matches, and removes anything
that does't match. For example the regex '^user.' will only keep User
extended attributes and anything else will be removed.
Mksquashfs also allows you to add extended attributes to files in the Squashfs
filesystem using the -xattrs-add option. This option takes an xattr name and
value pair separated by the '=' character.
The extended attribute name can be any valid name and can be in the namespaces
security, system, trusted, or user. User extended attributes are added to files
and directories (see man 7 xattr for explanation), and the others are added to
all files.
The extended attribute value by default will be treated as binary (i.e. an
uninterpreted byte sequence), but it can be prefixed with 0s, where it will be
treated as base64 encoded, or prefixed with 0x, where it will be treated as
hexidecimal.
Obviously using base64 or hexidecimal allows values to be used that cannot be
entered on the command line such as non-printable characters etc. But it
renders the string non-human readable. To keep readability and to allow
non-printable characters to be entered, the 0t prefix is supported. This
encoding is similar to binary encoding, except backslashes are specially
treated, and a backslash followed by three octal digits can be used to encode
any ASCII character, which obviously can be used to encode non-printable values.
The following four command lines are equivalent
mksquashfs dir image.sqfs -xattrs-add "user.comment=hello world"
mksquashfs dir image.sqfs -xattrs-add "user.comment=0saGVsbG8gd29ybGQ="
mksquashfs dir image.sqfs -xattrs-add "user.comment=0x68656c6c6f20776f726c64"
mksquashfs dir image.sqfs -xattrs-add "user.comment=0thello world"
Obviously in the above example there are no non-printable characters and so
the 0t prefixed string is identical to the first line. The following three
command lines are identical, but where the space has been replaced by the
non-printable NUL '\0' (null character).
mksquashfs dir image.sqfs -xattrs-add "user.comment=0thello\000world"
mksquashfs dir image.sqfs -xattrs-add "user.comment=0saGVsbG8Ad29ybGQ="
mksquashfs dir image.sqfs -xattrs-add "user.comment=0x68656c6c6f00776f726c64"
9. PSEUDO FILE SUPPORT
----------------------
Mksquashfs supports pseudo files, these allow files, directories, character
devices, block devices, fifos, symbolic links, hard links and extended
attributes to be specified and added to the Squashfs filesystem being built,
rather than requiring them to be present in the source files. This, for
example, allows device nodes to be added to the filesystem without requiring
root access.
Pseudo files also support "dynamic pseudo files" and a modify operation.
Dynamic pseudo files allow files to be dynamically created when Mksquashfs
is run, their contents being the result of running a command or piece of
shell script. The modify operation allows the mode/uid/gid of an existing
file in the source filesystem to be modified.
Two Mksquashfs options are supported, -p allows one pseudo file to be specified
on the command line, and -pf allows a pseudo file to be specified containing a
list of pseduo definitions, one per line.
9.1 CREATING A DYNAMIC FILE
---------------------------
Pseudo definition
Filename f mode uid gid command
mode is the octal mode specifier, similar to that expected by chmod.
uid and gid can be either specified as a decimal number, or by name.
command can be an executable or a piece of shell script, and it is executed
by running "/bin/sh -c command". The stdout becomes the contents of
"Filename".
Examples:
Running a basic command
-----------------------
/somedir/dmesg f 444 root root dmesg
creates a file "/somedir/dmesg" containing the output from dmesg.
Executing shell script
----------------------
RELEASE f 444 root root \
if [ ! -e /tmp/ver ]; then \
echo 0 > /tmp/ver; \
fi; \
ver=`cat /tmp/ver`; \
ver=$((ver +1)); \
echo $ver > /tmp/ver; \
echo -n `cat /tmp/release`; \
echo "-dev #"$ver `date` "Build host" `hostname`
Creates a file RELEASE containing the release name, date, build host, and
an incrementing version number. The incrementing version is a side-effect
of executing the shell script, and ensures every time Sqfstar is run a
new version number is used without requiring any other shell scripting.
The above example also shows that commands can be split across multiple lines
using "\". Obviously as the script will be presented to the shell as a single
line, a semicolon is need to separate individual shell commands within the
shell script.
Reading from a device (or fifo/named socket)
--------------------------------------------
input f 444 root root dd if=/dev/sda1 bs=1024 count=10
Copies 10K from the device /dev/sda1 into the file input. Ordinarily Sqfstar
given a device, fifo, or named socket will place that special file within the
Squashfs filesystem, the above allows input from these special files to be
captured and placed in the Squashfs filesystem.
9.2 CREATING A BLOCK OR CHARACTER DEVICE
----------------------------------------
Pseudo definition
Filename type mode uid gid major minor
Where type is either
b - for block devices, and
c - for character devices
mode is the octal mode specifier, similar to that expected by chmod.
uid and gid can be either specified as a decimal number, or by name.
For example:
/dev/chr_dev c 666 root root 100 1
/dev/blk_dev b 666 0 0 200 200
creates a character device "/dev/chr_dev" with major:minor 100:1 and
a block device "/dev/blk_dev" with major:minor 200:200, both with root
uid/gid and a mode of rw-rw-rw.
9.3 CREATING A DIRECTORY
-------------------------
Pseudo definition
Filename d mode uid gid
mode is the octal mode specifier, similar to that expected by chmod.
uid and gid can be either specified as a decimal number, or by name.
For example:
/pseudo_dir d 666 root root
creates a directory "/pseudo_dir" with root uid/gid and mode of rw-rw-rw.
9.4 CREATING A SYMBOLIC LINK
-----------------------------
Pseudo definition
Filename s mode uid gid symlink
uid and gid can be either specified as a decimal number, or by name.
Note mode is ignored, as symlinks always have "rwxrwxrwx" permissions.
For example:
symlink s 0 root root example
8reates a symlink "symlink" to file "example" with root uid/gid.
9.5 CREATING HARD LINKS (FILE REFERENCES)
-----------------------------------------
The "f" Pseudo definition allows a regular file to be created from the output of
a command (or shell). Often this is used to reference a file outside the source
directories by executing "cat", e.g.
README f 0555 0 0 cat /home/phillip/latest-version/README
Because this is a quite frequent use of the definition, an alternative faster
"File reference" or Hard Link Pseudo definition exists:
README l /home/phillip/latest-version/README
Will create a reference to "/home/phillip/latest-version/README",
and obviously the timestamp/mode and owership will be used.
The definition also be used to create additional references to
files within the source directories. For instance if "phillip/latest/README"
was a file being added to the filesystem, then
README l phillip/latest/README
Will create a Hard Link (and increment the nlink count on the inode).
In both cases, the path to the file being referenced is the system
filesystem path, and can be absolute (prefixed with /), or relative
to the current working directory.
There is an additional 'L' Pseudo definition, which closes a loophole in
the above 'l' definition. The 'l' Pseudo definition cannot create references
or Hard Links to files created by Pseudo definitions, because by
definition they do not exist in the system filesystem.
with 'L' the referenced file is expected to be a Pseudo file, and in this case
the path is taken to be from the root of the Squashfs filesystem being created,
e.g.
char-dev c 0555 0 0 1 2
link L char-dev
Will create a Hard Link named "link" to the character device called "char-dev"
created by the previous Pseudo definition.
9.6 CREATING SOCKETS/FIFOS
---------------------------
Pseudo definition
filename i mode uid gid [s|f]
To create a Unix domain socket, 's' should be used, i.e.
filename i 0777 root root s
and to create a FIFO, 'f' should be used, i.e.
filename i 0777 root root f
9.7 ADDING EXTENDED ATTRIBUTES TO FILES
---------------------------------------
Pseudo definition
filename x name=val
Will add the extended attribute <name> to <filename> with <val> contents. See
Section 7 for a description of the <val> formats supported.
9.8 MODIFYING ATTRIBUTES OF AN EXISTING FILE
--------------------------------------------
Pseudo definition
Filename m mode uid gid
mode is the octal mode specifier, similar to that expected by chmod.
uid and gid can be either specified as a decimal number, or by name.
For example:
dmesg m 666 root root
Changes the attributes of the file "dmesg" in the filesystem to have
root uid/gid and a mode of rw-rw-rw, overriding the attributes obtained
from the source filesystem.
10. EXTENDED PSEUDO FILE DEFINITIONS WITH TIMESTAMPS
----------------------------------------------------
The Pseudo file definitions described above do not allow the timestamp
of the created file to be specified, and the files will be timestamped
with the current time.
Extended versions of the Pseudo file definitions are supported which
take a <time> timestamp. These are distinquished from the previous
definitions by using an upper-case type character. For example the "D"
definition is identical to the "d" definition, but it takes a <time>
timestamp.
The list of extended definitions are:
filename F time mode uid gid command
filename D time mode uid gid
filename B time mode uid gid major minor
filename C time mode uid gid major minor
filename S time mode uid gid symlink
filename I time mode uid gid [s|f]
filename M time mode uid gid
<time> can be either an unsigned decimal integer (which represents the
seconds since the epoch of 1970-01-01 00:00 UTC), or a "date string"
which is parsed and converted into an integer since the epoch, by calling
the "date" command.
Because most date strings have spaces, they will need to be quoted, and if
entered on the command line, these quotes will need to be protected from the
shell by backslashes, i.e.
% mksquashfs dir image.sqsh -p "file D \"1 jan 1980\" 0777 phillip phillip"
Obviously anything "date" accepts as a valid string can be used, such as
"yesterday", "last week" etc.
11. APPENDING TO SQUASHFS FILESYSTEMS
-------------------------------------
Running Mksquashfs with the destination directory containing an existing
filesystem will add the source items to the existing filesystem. By default,
the source items are added to the existing root directory.
To make this clear... An existing filesystem "image" contains root entries
"old1", and "old2". Source directory "/home/phillip/test" contains "file1",
"file2" and "dir1".
example 1:
% mksquashfs /home/phillip/test image
Will create a new "image" with root entries "old1", "old2", "file1", "file2" and
"dir1"
example 2:
% mksquashfs /home/phillip/test image -keep-as-directory
Will create a new "image" with root entries "old1", "old2", and "test".
As shown in the previous section, for single source directories
'-keep-as-directory' adds the source directory rather than the
contents of the directory.
example 3:
% mksquashfs /home/phillip/test image -keep-as-directory -root-becomes
original-root
Will create a new "image" with root entries "original-root", and "test". The
'-root-becomes' option specifies that the original root becomes a subdirectory
in the new root, with the specified name.
The append option with file duplicate detection, means squashfs can be
used as a simple versioning archiving filesystem. A squashfs filesystem can
be created with for example the linux-2.4.19 source. Appending the linux-2.4.20
source will create a filesystem with the two source trees, but only the
changed files will take extra room, the unchanged files will be detected as
duplicates.
12. APPENDING RECOVERY FILE FEATURE
-----------------------------------
Recovery files are created when appending to existing Squashfs
filesystems. This allows the original filesystem to be recovered
if Mksquashfs aborts unexpectedly (i.e. power failure).
The recovery files are called squashfs_recovery_xxx_yyy, where
"xxx" is the name of the filesystem being appended to, and "yyy" is a
number to guarantee filename uniqueness (the PID of the parent Mksquashfs
process).
Normally if Mksquashfs exits correctly the recovery file is deleted to
avoid cluttering the filesystem. If Mksquashfs aborts, the "-recover"
option can be used to recover the filesystem, giving the previously
created recovery file as a parameter, i.e.
mksquashfs dummy image.sqsh -recover squashfs_recovery_image.sqsh_1234
The writing of the recovery file can be disabled by specifying the
"-no-recovery" option.
13. MKSQUASHFS ACTIONS INTRODUCTION
-----------------------------------
The Mksquashfs Actions code allows an "action" to be executed on a file if one
or more "tests" succeed. If you're familiar with the "find" command, then an
action is similar to "-print", and a test is similar to say "-name" or "-type".
To illustrate this it is useful to give two concrete examples.
example 1: the fragment action
% mksquashfs /home/phillip/github github.sqsh -action "fragment(cfiles) @ name(*.[ch])" -action "fragment(ofiles) @ name(*.o)"
This example defines two "fragment actions" which control the packing of files
within fragments. Specifically, it creates a specialised fragment called
"cfiles" which packs files matching the wildcard name "*.[ch]".
It also creates another specialised fragment called "ofiles" which packs files
matching the wilcard name "*.o".
Producing specialised fragments which only pack files which match a range of
tests, can produce better compression and/or I/O performance as it can optimise
similarity or access patterns. But it can also produce worse compression, and
so you should always test the effect.
Additionally, you should be able to see that an action definition is split into
an action function before the "@", and one or more test functions after the @.
Quoting is necessary here to protect it from interpretation by the shell. Also
the spacing before and after the "@" isn't necessary and is used here for added
readability.
example 2: the uncompressed action
% mksquashfs /home/phillip backup.sqsh -action "uncompressed @ ( name(*.jpg) || name(*.mpg) ) || ( name(*.img) && filesize(+1G) )"
This is a more complex example. It tells Mksquashfs to not try and compress any
file which matches the wildcard names "*.jpg" and "*.mpg". But it also tells
Mksquashfs not to try and compress files which match the wildcard name "*.img"
and are also 1 Gigabyte in size or larger.
This example introduces the fact that tests can be combined using the logical
operators && (and), || (or) and ! (not), and can be bracketed.
Please see the ACTIONS-README file for syntax and extra information.
14. MISCELLANEOUS OPTIONS
------------------------
The -info option displays the files/directories as they are compressed and
added to the filesystem. The original uncompressed size of each file
is printed, along with DUPLICATE if the file is a duplicate of a
file in the filesystem.
The -nopad option informs Mksquashfs to not pad the filesystem to a 4K multiple.
This is performed by default to enable the output filesystem file to be mounted
by loopback, which requires files to be a 4K multiple. If the filesystem is
being written to a block device, or is to be stored in a bootimage, the extra
pad bytes are not needed.
|