If using arduino environment and wifi manager to manage connections then the fast connect can be handled automatically by using the feature_fastconnect version of the library. This retrieves the channel / mac address when first making a connection and then subsequently uses them. It also handles the changes needed if the channel or mac address of the router change.
Perhaps if u try to use only esp-idf and after make it work try to shift that knowledge to arduino core????