What about Inputting Policy in Value Function: Policy Representation and Policy-Extended Value Function Approximator.

Hongyao Tang, Zhaopeng Meng, Jianye Hao, Chen Chen, Daniel Graves, Dong Li, Changmin Yu, Hangyu Mao, Wulong Liu, Yaodong Yang, Wenyuan Tao, Li Wang

Browse the full AAAI paper archive.