*** buffer overflow detected ***: terminated error message when using find . of android phone

Buffer overflow: It looks like zip is a bit picky about file naming:

zip /var/tmp/a.zip 02\ -\ Why\ Can’t\ the\ English\?.mp3 
*** buffer overflow detected ***: terminated


zip error: Interrupted (aborting)

cp 02\ -\ Why\ Can’t\ the\ English\?.mp3 02-WhyCanttheEnglish.mp3

zip /var/tmp/a.zip 02-WhyCanttheEnglish.mp3 
  adding: 02-WhyCanttheEnglish.mp3 (deflated 1%)


I’ve no idea yet what the reason is for zip to break out at this point, it’s not only the quote or question mark. Needs debugging.

Little debugging: routines local_to_wide string and local_to_utf8 string are involved, so
it has to do with the unusual characters in the file name.

Yes my point exacly, if you are having the same problem creating a zip file from a simple file, there has to be something else that is the problem. Also, what are all those slashes for? Have you tested it on a webdav mount so any problems correlate as closely as possible to mine? if you use zip -r -0 (no compression) does it make a difference? Why does a buffer overflow get detected when u are just zipping one file? In the second example, you just copied and renamed the original source file am I correct? What is your working directory?

It’s simple: the file name is read in and the file is opened correctly in zip. Then the name gets processed somehow for storage in the zip file. Either the character set of the file name is not recognized properly, or the conversion fails somehow. So it’s a bug in zip or in the libraries called by zip, where the file is coming from is not important.
The backslashes are escape characters, file names with spaces have to be
“file with space”, ‘file with space’, or file\ with\ space.

I can create a simple zip file when im on local storage using both the find & zip command together if the source files are stored locally. The problem arises when source files are on webdav and then saving into local storage.

I’m trying to debug but I’m afraid I’ve to give up. SELinux is off.

zip /var/tmp/a.zip 02\ -\ Why\ Can’t\ the\ English\?.mp3 
*** buffer overflow detected ***: terminated


zip error: Interrupted (aborting)
zip /tmp/a.zip 02\ -\ Why\ Can’t\ the\ English\?.mp3 
updating: 02 - Why Can’t the English?.mp3 (deflated 1%)

What’s going on here??? Destination directory determines what happens?

Edit: no magic, every location is wrong except if the zip file with that name already exists.

https://bugzilla.redhat.com/show_bug.cgi?id=2165653

Bug in zip. Maybe a side effect of recent build flag changes (the buffer overflow bug was always there, but now it’s exposed by stricter checks?).

Zip treats white space as a break between file names, so it cannot find that file. The shell also treats the ' as a special character and expects a closing quote with it.
If one were to enclose the full file name with single quotes it might work since zip may then see the quoted name as a single file name. (I have not tested this)

White space and special characters have been avoided from the beginning in unix-like OSes since a white space has always been considered a file name break and command line word separators.
All one needs to do to recognize the difference is to use the ls command in a directory where some files have white space or special characters. Those file names will appear with quotes when they are displayed.

A partial explanation is shown here, but does not cover all cases with all apps.

edit:
I just tested with the file being zipped having white space in the name and enclosing the file name in single quotes and it worked.

$ zip add 'Band Chart 8_5 X 11 Color.pdf'
  adding: Band Chart 8_5 X 11 Color.pdf (deflated 24%)

This created the file add.zip containing the named file.

$ zip add2 Band\ Chart\ 8_5\ X\ 11\ Color.pdf
  adding: Band Chart 8_5 X 11 Color.pdf (deflated 24%)

This also worked with every special character escaped and created the file ‘add2.zip’.
Special characters include '(){}[],$;"\.? and the white space characters with some I am sure I missed.

Wrong; the spaces are escaped. foo\ bar is the same as 'foo bar':

$ touch foo\ bar
$ zip foo.zip foo\ bar # no error

The error is from certain characters, e.g. (right single quotation mark):

$ touch foo’
$ zip foo.zip foo’
*** buffer overflow detected ***: terminated


zip error: Interrupted (aborting)

Please see the linked bug for more info.

I don’t think that is a bug since the shell interprets the single quote before it gets to zip. A single quote must be escaped before it is treated as a literal character.

right single quotation mark (sometimes called “curly quote”), not ' apostrophe (aka “single quote”).

If it was an unmatched single quote, it would open a secondary prompt to continue input:

$ touch foo'
>

until you close the quote.

Yes, and I showed that just above.

But that was not escaped.

okay it seems we are getting off-topic but are you guys saying the reason for the buffer overflow error is because there are unescaped white spaces or special characters in the filename ? How can you explain why a normal filename such as 20230531_215584.jpg not being zipped?

The bug is not related to spaces, single quotes, special characters (in bash terms, like '"?), or escaping or not escaping characters. It’s related to the length of certain Unicode characters (like ’á©) being wrongly calculated.

I’m not sure why @computersavvy started speculating about spaces, quoting, and escaping, when the exact problem in the code was already identified.

@hmmsjan was also very close to identifying it and mentioned utf8 characters.

I don’t know. Do you have an example? Did you copy the file off the phone first, to narrow down the problem to zip and not something about gvfs or other factors?

What do you mean it is off?

what are your destination directories what locations?

Yes, as a matter of fact here is the test using a sample file that failed to zip with gvfs – 20220731_180105.jpg


So, to copy the same file off from the webdav server mount into local storage then zipping the file is successful.

Now, when I try to copy multiple files using cp -p -r instead of zipping them I am still getting alot of input/output errors just like zip I/O errors

OK Found, I’ll report it in the bug report above.
Edit: overseen, the bug report has already a patch added, exactly the same.

Nice bugs with things you hardly see: no quote but special version of quote taking 3 bytes. Thanks to my wife’s My Fair Lady CD on my disk…

In file fileio.c:
3505 wsize = mbstowcs(wc_string, local_string, strlen(local_string) + 1);

the local_string in UTF-8 is converted to double byte characters, the max number is in the last argument. The size of the wc_string is calculated before, but the author derives the number of characters from the original string, which is mostly fine, but the actual redhat way of compiling includes a bound check and checks that wc_string is big enough. Result: Failure.
The overflow error does not belong to zip, but the bound checking in the libraries, so it could pop up in other points in the program !!! At least this error can be fixed in the next release

Patch:

--- zip30/fileio.c.orig 2023-05-31 23:15:01.875039323 +0200
+++ zip30/fileio.c      2023-05-31 23:16:00.976291515 +0200
@@ -3502,7 +3502,7 @@
   if ((wc_string = (wchar_t *)malloc((wsize + 1) * sizeof(wchar_t))) == NULL) {
     ZIPERR(ZE_MEM, "local_to_wide_string");
   }
-  wsize = mbstowcs(wc_string, local_string, strlen(local_string) + 1);
+  wsize = mbstowcs(wc_string, local_string, wsize + 1);
   wc_string[wsize] = (wchar_t) 0;

   /* in case wchar_t is not zwchar */

1 Like

I know that I posted ubuntu vmware screenshots but the same kind of errors are also on fedora

So is this to correct the buffer overflow detected error ? How can I change that code and what file is it ? Have you tested this to make sure it works?

I’ve sent a PM, never did this before so hope this works.

Thank you for the help it seems like the buffer overflow error is not showing up so far… however, now I am getting

zip I/O error: Bad address
zip error: Output file write failure (write error on zip file)

here is the linking post: Fedora 38 - I/O zipping error using find command - #48 by sixpack13

lol this never ends…