When "not overwrite" is specified, ao_open_file() will try to first open it for reading and this does not fail, open it for writing.

if (!overwrite) {
  /* Test for file existence */
  file = fopen(filename, "r");
  if (file != NULL) {
    fclose(file);
    errno = AO_EFILEEXISTS;
    return NULL;
  }
}


file = fopen(filename, "w");

There is a TOCTTOU condition, the file could have been created by someone else after the check.

A non-racy way to create it would be to use

file=fopen(filename, "wx");  

and checking for the errno E_EXIST.

If this is not portable enough, open( ... , O_CREAT|O_EXCL) could be used.