Change the file time before we change the file mode.