When looking at the implementation of device_reference and tuple_of_iterator_references, I discovered that the swap support may be unnecessarily constraining (compared to _BitReference from libstdc++, which I originally took as a reference for how a reference-like type could be implemented):

using std::swap;
thrust::device_vector<int> vec(4);
int i{};
auto ref = vec[0];
auto ref2 = vec[1];
swap(vec[0], vec[1]); // 1
swap(ref, ref2);      // 1*
swap(i, vec[0]);      // 2
swap(vec[0], i);      // 3

auto it = thrust::make_zip_iterator(vec.begin(), vec.begin() + 1);
thrust::tuple<int, int> tuple;
swap(it[0], it[2]); // 4
swap(it[0], tuple); // 5
swap(tuple, it[0]); // 6

https://godbolt.org/z/4rbK3WhM3

1 does not work (but is intended to work, I guess?) because the swap specialization for device_reference unnecessarily (?) requires the input to be a mutable l-value reference instead of just a value, as 1* shows.
2 and 3 do not have associated specializations at all, same as 5 and 6.

This should probably be easy to fix: Add the 4 missing specializations and remove the reference qualifier.

EDIT: On second thought, adding 5 and 6 is probably highly non-trivial, as there seems to be no existing way to get the value type from a reference-like object? And to truly tackle 2 and 3, one would need to understand the details of tagged_reference.