Yes my point exacly, if you are having the same problem creating a zip file from a simple file, there has to be something else that is the problem. Also, what are all those slashes for? Have you tested it on a webdav mount so any problems correlate as closely as possible to mine? if you use zip -r -0 (no compression) does it make a difference? Why does a buffer overflow get detected when u are just zipping one file? In the second example, you just copied and renamed the original source file am I correct? What is your working directory?
It’s simple: the file name is read in and the file is opened correctly in zip. Then the name gets processed somehow for storage in the zip file. Either the character set of the file name is not recognized properly, or the conversion fails somehow. So it’s a bug in zip or in the libraries called by zip, where the file is coming from is not important.
The backslashes are escape characters, file names with spaces have to be
“file with space”, ‘file with space’, or file\ with\ space.
I can create a simple zip file when im on local storage using both the find & zip command together if the source files are stored locally. The problem arises when source files are on webdav and then saving into local storage.
I’m trying to debug but I’m afraid I’ve to give up. SELinux is off.
zip /var/tmp/a.zip 02\ -\ Why\ Can’t\ the\ English\?.mp3
*** buffer overflow detected ***: terminated
zip error: Interrupted (aborting)
zip /tmp/a.zip 02\ -\ Why\ Can’t\ the\ English\?.mp3
updating: 02 - Why Can’t the English?.mp3 (deflated 1%)
What’s going on here??? Destination directory determines what happens?
Edit: no magic, every location is wrong except if the zip file with that name already exists.
https://bugzilla.redhat.com/show_bug.cgi?id=2165653
Bug in zip. Maybe a side effect of recent build flag changes (the buffer overflow bug was always there, but now it’s exposed by stricter checks?).
Zip treats white space as a break between file names, so it cannot find that file. The shell also treats the ' as a special character and expects a closing quote with it.
If one were to enclose the full file name with single quotes it might work since zip may then see the quoted name as a single file name. (I have not tested this)
White space and special characters have been avoided from the beginning in unix-like OSes since a white space has always been considered a file name break and command line word separators.
All one needs to do to recognize the difference is to use the ls command in a directory where some files have white space or special characters. Those file names will appear with quotes when they are displayed.
A partial explanation is shown here, but does not cover all cases with all apps.
edit:
I just tested with the file being zipped having white space in the name and enclosing the file name in single quotes and it worked.
$ zip add 'Band Chart 8_5 X 11 Color.pdf'
adding: Band Chart 8_5 X 11 Color.pdf (deflated 24%)
This created the file add.zip containing the named file.
$ zip add2 Band\ Chart\ 8_5\ X\ 11\ Color.pdf
adding: Band Chart 8_5 X 11 Color.pdf (deflated 24%)
This also worked with every special character escaped and created the file ‘add2.zip’.
Special characters include '(){}[],$;"\.? and the white space characters with some I am sure I missed.
Wrong; the spaces are escaped. foo\ bar is the same as 'foo bar':
$ touch foo\ bar
$ zip foo.zip foo\ bar # no error
The error is from certain characters, e.g. ’ (right single quotation mark):
$ touch foo’
$ zip foo.zip foo’
*** buffer overflow detected ***: terminated
zip error: Interrupted (aborting)
Please see the linked bug for more info.
I don’t think that is a bug since the shell interprets the single quote before it gets to zip. A single quote must be escaped before it is treated as a literal character.
’ right single quotation mark (sometimes called “curly quote”), not ' apostrophe (aka “single quote”).
If it was an unmatched single quote, it would open a secondary prompt to continue input:
$ touch foo'
>
until you close the quote.
Yes, and I showed that just above.
But that was not escaped.
okay it seems we are getting off-topic but are you guys saying the reason for the buffer overflow error is because there are unescaped white spaces or special characters in the filename ? How can you explain why a normal filename such as 20230531_215584.jpg not being zipped?
The bug is not related to spaces, single quotes, special characters (in bash terms, like '"?), or escaping or not escaping characters. It’s related to the length of certain Unicode characters (like ’á©) being wrongly calculated.
I’m not sure why @computersavvy started speculating about spaces, quoting, and escaping, when the exact problem in the code was already identified.
@hmmsjan was also very close to identifying it and mentioned utf8 characters.
I don’t know. Do you have an example? Did you copy the file off the phone first, to narrow down the problem to zip and not something about gvfs or other factors?
What do you mean it is off?
what are your destination directories what locations?
Yes, as a matter of fact here is the test using a sample file that failed to zip with gvfs – 20220731_180105.jpg
So, to copy the same file off from the webdav server mount into local storage then zipping the file is successful.
Now, when I try to copy multiple files using cp -p -r instead of zipping them I am still getting alot of input/output errors just like zip I/O errors
OK Found, I’ll report it in the bug report above.
Edit: overseen, the bug report has already a patch added, exactly the same.
Nice bugs with things you hardly see: no quote but special version of quote taking 3 bytes. Thanks to my wife’s My Fair Lady CD on my disk…
In file fileio.c:
3505 wsize = mbstowcs(wc_string, local_string, strlen(local_string) + 1);
the local_string in UTF-8 is converted to double byte characters, the max number is in the last argument. The size of the wc_string is calculated before, but the author derives the number of characters from the original string, which is mostly fine, but the actual redhat way of compiling includes a bound check and checks that wc_string is big enough. Result: Failure.
The overflow error does not belong to zip, but the bound checking in the libraries, so it could pop up in other points in the program !!! At least this error can be fixed in the next release
Patch:
--- zip30/fileio.c.orig 2023-05-31 23:15:01.875039323 +0200
+++ zip30/fileio.c 2023-05-31 23:16:00.976291515 +0200
@@ -3502,7 +3502,7 @@
if ((wc_string = (wchar_t *)malloc((wsize + 1) * sizeof(wchar_t))) == NULL) {
ZIPERR(ZE_MEM, "local_to_wide_string");
}
- wsize = mbstowcs(wc_string, local_string, strlen(local_string) + 1);
+ wsize = mbstowcs(wc_string, local_string, wsize + 1);
wc_string[wsize] = (wchar_t) 0;
/* in case wchar_t is not zwchar */
1 Like
I know that I posted ubuntu vmware screenshots but the same kind of errors are also on fedora
So is this to correct the buffer overflow detected error ? How can I change that code and what file is it ? Have you tested this to make sure it works?
I’ve sent a PM, never did this before so hope this works.
Thank you for the help it seems like the buffer overflow error is not showing up so far… however, now I am getting
zip I/O error: Bad address
zip error: Output file write failure (write error on zip file)
here is the linking post: Fedora 38 - I/O zipping error using find command - #48 by sixpack13
lol this never ends…
Great, that’s one problem solved, but this is the original one, and to debug one needs the circumstances you have.
Edit: The things below will probably not help you.
A: you do not have the debug info for the patched version
B: It might be an error trapped by the program itself with the risk that it prints the error message and exits normally seen from the debugger.
There is one thing you can do, hope that there is some time between start of zip and the occurrence of the error.
Prerequisite: gdb is installed.
Open two terminal windows.
Start the PIPED command I suggested including "| zip -0 -@ "
in one window
As soon as possible in the other window:
pgrep zip
It shows you the process number of zip. Alternative: “ps ax | grep zip”
gdb -p theprocessnumber
Answer yes if it wants to download something a few times.
type “c” if it shows some output indicating that it is waiting
Wait until the error. zip can be slower now.
gdb now shows some output
type “bt” and post the output.
At least it may be possible to get the location of the error.
Example session, where zip waits for input on stdin and is killed by killall
pgrep zip
7208
[hj@lap2 ~]$ gdb -p 7208
GNU gdb (GDB) Fedora Linux 13.1-4.fc38
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 7208
Reading symbols from /usr/bin/zip...
This GDB supports auto-downloading debuginfo from the following URLs:
<https://debuginfod.fedoraproject.org/>
Enable debuginfod for this session? (y or [n]) y
Debuginfod has been enabled.
To make this setting permanent, add 'set debuginfod enabled on' to .gdbinit.
Downloading separate debug info for /usr/bin/zip
Reading symbols from .gnu_debugdata for /usr/bin/zip...
(No debugging symbols found in .gnu_debugdata for /usr/bin/zip)
Reading symbols from /lib64/libbz2.so.1...
Reading symbols from /home/hj/.cache/debuginfod_client/f8237ff24b622047428b97b1486b393e3a9e38de/debuginfo...
Reading symbols from /lib64/libc.so.6...
Reading symbols from /home/hj/.cache/debuginfod_client/245240a31888ad5c11bbc55b18e02d87388f59a9/debuginfo...
Reading symbols from /lib64/ld-linux-x86-64.so.2...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f3e2d2d70c1 in __GI___libc_read (fd=0, buf=0x1ccd410, nbytes=1024)
at ../sysdeps/unix/sysv/linux/read.c:26
26 return SYSCALL_CANCEL (read, fd, buf, nbytes);
--Type <RET> for more, q to quit, c to continue without paging--
Missing separate debuginfos, use: dnf debuginfo-install zip-3.0-36.post1.fc38.x86_64
(gdb) c
Continuing.
Program received signal SIGTERM, Terminated.
0x00007f3e2d2d70c1 in __GI___libc_read (fd=0, buf=0x1ccd410, nbytes=1024)
at ../sysdeps/unix/sysv/linux/read.c:26
26 return SYSCALL_CANCEL (read, fd, buf, nbytes);
(gdb) bt
#0 0x00007f3e2d2d70c1 in __GI___libc_read (fd=0, buf=0x1ccd410, nbytes=1024)
at ../sysdeps/unix/sysv/linux/read.c:26
#1 0x00007f3e2d25ab7b in _IO_new_file_underflow (
fp=0x7f3e2d3aaaa0 <_IO_2_1_stdin_>)
at /usr/src/debug/glibc-2.37-4.fc38.x86_64/libio/libioP.h:946
#2 0x00007f3e2d25bf79 in __GI__IO_default_uflow (
fp=0x7f3e2d3aaaa0 <_IO_2_1_stdin_>)
at /usr/src/debug/glibc-2.37-4.fc38.x86_64/libio/libioP.h:946
#3 0x0000000000412cc8 in getnam ()
#4 0x00000000004064e7 in main ()
(gdb) quit