The Inspector General's report calls for better oversight, but one official said his office isn't responsible.